2023-03-28 20:53:30,126 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda 2023-03-28 20:53:30,140 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-28 20:53:30,154 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-28 20:53:30,155 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f, deleteOnExit=true 2023-03-28 20:53:30,155 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-28 20:53:30,156 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/test.cache.data in system properties and HBase conf 2023-03-28 20:53:30,156 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/hadoop.tmp.dir in system properties and HBase conf 2023-03-28 20:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/hadoop.log.dir in system properties and HBase conf 2023-03-28 20:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-28 20:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-28 20:53:30,158 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-28 20:53:30,275 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-28 20:53:30,705 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-28 20:53:30,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-28 20:53:30,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-28 20:53:30,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-28 20:53:30,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-28 20:53:30,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-28 20:53:30,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-28 20:53:30,714 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-28 20:53:30,714 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-28 20:53:30,714 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-28 20:53:30,715 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/nfs.dump.dir in system properties and HBase conf 2023-03-28 20:53:30,715 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/java.io.tmpdir in system properties and HBase conf 2023-03-28 20:53:30,715 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-28 20:53:30,716 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-28 20:53:30,716 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-28 20:53:31,328 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-28 20:53:31,333 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-28 20:53:31,632 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-28 20:53:31,816 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-28 20:53:31,833 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-28 20:53:31,873 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-28 20:53:31,908 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/java.io.tmpdir/Jetty_localhost_41285_hdfs____ovvnb0/webapp 2023-03-28 20:53:32,052 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41285 2023-03-28 20:53:32,068 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-28 20:53:32,068 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-28 20:53:32,629 WARN [Listener at localhost/34423] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-28 20:53:32,710 WARN [Listener at localhost/34423] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-28 20:53:32,730 WARN [Listener at localhost/34423] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-28 20:53:32,737 INFO [Listener at localhost/34423] log.Slf4jLog(67): jetty-6.1.26 2023-03-28 20:53:32,743 INFO [Listener at localhost/34423] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/java.io.tmpdir/Jetty_localhost_43269_datanode____ev8j25/webapp 2023-03-28 20:53:32,844 INFO [Listener at localhost/34423] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43269 2023-03-28 20:53:33,255 WARN [Listener at localhost/34739] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-28 20:53:33,772 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xca7a28075e06474d: Processing first storage report for DS-4916f720-d31e-4004-bdf3-8c8f70bf3833 from datanode e8b38ec5-35e4-41ef-a012-5de0dd603912 2023-03-28 20:53:33,774 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xca7a28075e06474d: from storage DS-4916f720-d31e-4004-bdf3-8c8f70bf3833 node DatanodeRegistration(127.0.0.1:36815, datanodeUuid=e8b38ec5-35e4-41ef-a012-5de0dd603912, infoPort=33423, infoSecurePort=0, ipcPort=34739, storageInfo=lv=-57;cid=testClusterID;nsid=1250494567;c=1680036811406), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-28 20:53:33,774 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xca7a28075e06474d: Processing first storage report for DS-f41fe2e1-1a3b-4375-a3bf-f374fe39135d from datanode e8b38ec5-35e4-41ef-a012-5de0dd603912 2023-03-28 20:53:33,774 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xca7a28075e06474d: from storage DS-f41fe2e1-1a3b-4375-a3bf-f374fe39135d node DatanodeRegistration(127.0.0.1:36815, datanodeUuid=e8b38ec5-35e4-41ef-a012-5de0dd603912, infoPort=33423, infoSecurePort=0, ipcPort=34739, storageInfo=lv=-57;cid=testClusterID;nsid=1250494567;c=1680036811406), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-28 20:53:33,784 DEBUG [Listener at localhost/34739] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda 2023-03-28 20:53:33,847 INFO [Listener at localhost/34739] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f/zookeeper_0, clientPort=56827, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-28 20:53:33,861 INFO [Listener at localhost/34739] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=56827 2023-03-28 20:53:33,871 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:33,873 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:34,541 INFO [Listener at localhost/34739] util.FSUtils(471): Created version file at hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec with version=8 2023-03-28 20:53:34,541 INFO [Listener at localhost/34739] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/hbase-staging 2023-03-28 20:53:34,867 INFO [Listener at localhost/34739] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-28 20:53:35,373 INFO [Listener at localhost/34739] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-28 20:53:35,409 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,409 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,410 INFO [Listener at localhost/34739] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-28 20:53:35,410 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,410 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-28 20:53:35,572 INFO [Listener at localhost/34739] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-28 20:53:35,660 DEBUG [Listener at localhost/34739] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-28 20:53:35,755 INFO [Listener at localhost/34739] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44521 2023-03-28 20:53:35,766 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:35,768 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:35,789 INFO [Listener at localhost/34739] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44521 connecting to ZooKeeper ensemble=127.0.0.1:56827 2023-03-28 20:53:35,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:445210x0, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-28 20:53:35,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44521-0x1012ce745ce0000 connected 2023-03-28 20:53:35,861 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-28 20:53:35,862 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-28 20:53:35,865 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-28 20:53:35,873 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44521 2023-03-28 20:53:35,873 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44521 2023-03-28 20:53:35,874 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44521 2023-03-28 20:53:35,874 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44521 2023-03-28 20:53:35,875 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44521 2023-03-28 20:53:35,882 INFO [Listener at localhost/34739] master.HMaster(444): hbase.rootdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec, hbase.cluster.distributed=false 2023-03-28 20:53:35,957 INFO [Listener at localhost/34739] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-28 20:53:35,957 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,957 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,958 INFO [Listener at localhost/34739] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-28 20:53:35,958 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-28 20:53:35,958 INFO [Listener at localhost/34739] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-28 20:53:35,963 INFO [Listener at localhost/34739] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-28 20:53:35,965 INFO [Listener at localhost/34739] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42693 2023-03-28 20:53:35,968 INFO [Listener at localhost/34739] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-28 20:53:35,974 DEBUG [Listener at localhost/34739] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-28 20:53:35,975 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:35,978 INFO [Listener at localhost/34739] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:35,980 INFO [Listener at localhost/34739] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42693 connecting to ZooKeeper ensemble=127.0.0.1:56827 2023-03-28 20:53:35,984 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:426930x0, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-28 20:53:35,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42693-0x1012ce745ce0001 connected 2023-03-28 20:53:35,985 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-28 20:53:35,986 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-28 20:53:35,987 DEBUG [Listener at localhost/34739] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-28 20:53:35,992 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42693 2023-03-28 20:53:35,992 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42693 2023-03-28 20:53:35,992 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42693 2023-03-28 20:53:35,995 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42693 2023-03-28 20:53:35,995 DEBUG [Listener at localhost/34739] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42693 2023-03-28 20:53:35,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:36,009 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-28 20:53:36,011 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-28 20:53:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-28 20:53:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:36,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-28 20:53:36,033 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44521,1680036814691 from backup master directory 2023-03-28 20:53:36,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-28 20:53:36,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:36,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-28 20:53:36,037 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-28 20:53:36,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:36,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-28 20:53:36,042 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-28 20:53:36,531 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/hbase.id with ID: 3054be00-513e-4a32-99b1-77af66c52820 2023-03-28 20:53:36,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-28 20:53:36,583 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:36,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x063e7276 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:36,653 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30fe1cb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:36,676 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:53:36,678 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-28 20:53:36,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-28 20:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-28 20:53:36,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:36,700 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:36,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-28 20:53:36,735 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store-tmp 2023-03-28 20:53:36,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:36,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-28 20:53:36,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:53:36,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:53:36,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-28 20:53:36,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:53:36,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:53:36,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-28 20:53:36,764 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/WALs/jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:36,787 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44521%2C1680036814691, suffix=, logDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/WALs/jenkins-hbase4.apache.org,44521,1680036814691, archiveDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/oldWALs, maxLogs=10 2023-03-28 20:53:36,839 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK] 2023-03-28 20:53:36,847 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:36,911 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/WALs/jenkins-hbase4.apache.org,44521,1680036814691/jenkins-hbase4.apache.org%2C44521%2C1680036814691.1680036816797 2023-03-28 20:53:36,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK]] 2023-03-28 20:53:36,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:53:36,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:36,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:36,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:36,992 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:37,000 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-28 20:53:37,029 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-28 20:53:37,042 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:37,049 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:37,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:37,065 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-28 20:53:37,073 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:53:37,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61248315, jitterRate=-0.08732898533344269}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-28 20:53:37,075 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-28 20:53:37,076 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-28 20:53:37,102 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-28 20:53:37,102 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-28 20:53:37,104 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-28 20:53:37,106 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-28 20:53:37,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 42 msec 2023-03-28 20:53:37,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-28 20:53:37,183 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-28 20:53:37,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-28 20:53:37,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-28 20:53:37,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-28 20:53:37,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-28 20:53:37,224 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-28 20:53:37,228 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-28 20:53:37,232 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:37,233 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-28 20:53:37,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-28 20:53:37,247 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-28 20:53:37,253 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-28 20:53:37,253 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-28 20:53:37,253 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:37,254 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44521,1680036814691, sessionid=0x1012ce745ce0000, setting cluster-up flag (Was=false) 2023-03-28 20:53:37,271 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:37,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-28 20:53:37,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:37,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:37,291 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-28 20:53:37,293 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:37,296 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.hbase-snapshot/.tmp 2023-03-28 20:53:37,301 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(952): ClusterId : 3054be00-513e-4a32-99b1-77af66c52820 2023-03-28 20:53:37,307 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-28 20:53:37,314 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-28 20:53:37,314 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-28 20:53:37,321 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-28 20:53:37,322 DEBUG [RS:0;jenkins-hbase4:42693] zookeeper.ReadOnlyZKClient(139): Connect 0x086c7f07 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:37,328 DEBUG [RS:0;jenkins-hbase4:42693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ab807a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:37,328 DEBUG [RS:0;jenkins-hbase4:42693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2941477, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-28 20:53:37,355 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42693 2023-03-28 20:53:37,358 INFO [RS:0;jenkins-hbase4:42693] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-28 20:53:37,358 INFO [RS:0;jenkins-hbase4:42693] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-28 20:53:37,359 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1023): About to register with Master. 2023-03-28 20:53:37,362 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44521,1680036814691 with isa=jenkins-hbase4.apache.org/172.31.14.131:42693, startcode=1680036815956 2023-03-28 20:53:37,379 DEBUG [RS:0;jenkins-hbase4:42693] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-28 20:53:37,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-28 20:53:37,441 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49211, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-28 20:53:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:37,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-28 20:53:37,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-28 20:53:37,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-28 20:53:37,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-28 20:53:37,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-28 20:53:37,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-28 20:53:37,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680036847456 2023-03-28 20:53:37,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-28 20:53:37,460 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-28 20:53:37,461 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-28 20:53:37,467 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-28 20:53:37,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-28 20:53:37,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-28 20:53:37,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-28 20:53:37,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-28 20:53:37,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-28 20:53:37,483 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(2829): Master is not running yet 2023-03-28 20:53:37,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,484 WARN [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-28 20:53:37,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-28 20:53:37,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-28 20:53:37,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-28 20:53:37,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-28 20:53:37,490 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-28 20:53:37,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680036817492,5,FailOnTimeoutGroup] 2023-03-28 20:53:37,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680036817492,5,FailOnTimeoutGroup] 2023-03-28 20:53:37,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-28 20:53:37,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,505 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-28 20:53:37,506 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-28 20:53:37,507 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec 2023-03-28 20:53:37,527 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:37,529 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-28 20:53:37,532 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/info 2023-03-28 20:53:37,534 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-28 20:53:37,535 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:37,535 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-28 20:53:37,539 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/rep_barrier 2023-03-28 20:53:37,540 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-28 20:53:37,542 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:37,542 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-28 20:53:37,545 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/table 2023-03-28 20:53:37,546 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-28 20:53:37,547 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:37,549 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740 2023-03-28 20:53:37,550 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740 2023-03-28 20:53:37,554 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:53:37,556 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-28 20:53:37,562 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:53:37,563 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66104604, jitterRate=-0.014964640140533447}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:53:37,563 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-28 20:53:37,563 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-28 20:53:37,563 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-28 20:53:37,563 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-28 20:53:37,563 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-28 20:53:37,563 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-28 20:53:37,568 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-28 20:53:37,568 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-28 20:53:37,574 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-28 20:53:37,574 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-28 20:53:37,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-28 20:53:37,585 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44521,1680036814691 with isa=jenkins-hbase4.apache.org/172.31.14.131:42693, startcode=1680036815956 2023-03-28 20:53:37,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,598 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-28 20:53:37,599 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec 2023-03-28 20:53:37,599 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34423 2023-03-28 20:53:37,599 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-28 20:53:37,602 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-28 20:53:37,604 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-28 20:53:37,606 DEBUG [RS:0;jenkins-hbase4:42693] zookeeper.ZKUtil(162): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,606 WARN [RS:0;jenkins-hbase4:42693] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-28 20:53:37,606 INFO [RS:0;jenkins-hbase4:42693] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-28 20:53:37,606 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/WALs/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,607 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42693,1680036815956] 2023-03-28 20:53:37,614 DEBUG [RS:0;jenkins-hbase4:42693] zookeeper.ZKUtil(162): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,625 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-28 20:53:37,633 INFO [RS:0;jenkins-hbase4:42693] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-28 20:53:37,658 INFO [RS:0;jenkins-hbase4:42693] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-28 20:53:37,663 INFO [RS:0;jenkins-hbase4:42693] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-28 20:53:37,663 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,664 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-28 20:53:37,672 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-28 20:53:37,673 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,674 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,674 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,674 DEBUG [RS:0;jenkins-hbase4:42693] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-28 20:53:37,675 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,675 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,675 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,691 INFO [RS:0;jenkins-hbase4:42693] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-28 20:53:37,693 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42693,1680036815956-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:37,716 INFO [RS:0;jenkins-hbase4:42693] regionserver.Replication(203): jenkins-hbase4.apache.org,42693,1680036815956 started 2023-03-28 20:53:37,716 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42693,1680036815956, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42693, sessionid=0x1012ce745ce0001 2023-03-28 20:53:37,716 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-28 20:53:37,716 DEBUG [RS:0;jenkins-hbase4:42693] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,716 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42693,1680036815956' 2023-03-28 20:53:37,716 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:37,717 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42693,1680036815956' 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-28 20:53:37,718 DEBUG [RS:0;jenkins-hbase4:42693] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-28 20:53:37,719 DEBUG [RS:0;jenkins-hbase4:42693] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-28 20:53:37,719 INFO [RS:0;jenkins-hbase4:42693] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-28 20:53:37,719 INFO [RS:0;jenkins-hbase4:42693] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-28 20:53:37,754 DEBUG [jenkins-hbase4:44521] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-28 20:53:37,757 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42693,1680036815956, state=OPENING 2023-03-28 20:53:37,767 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-28 20:53:37,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:37,770 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-28 20:53:37,773 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:53:37,830 INFO [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42693%2C1680036815956, suffix=, logDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/WALs/jenkins-hbase4.apache.org,42693,1680036815956, archiveDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/oldWALs, maxLogs=32 2023-03-28 20:53:37,849 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK] 2023-03-28 20:53:37,852 INFO [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/WALs/jenkins-hbase4.apache.org,42693,1680036815956/jenkins-hbase4.apache.org%2C42693%2C1680036815956.1680036817833 2023-03-28 20:53:37,852 DEBUG [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK]] 2023-03-28 20:53:37,958 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:37,960 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-28 20:53:37,964 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38704, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-28 20:53:37,975 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-28 20:53:37,975 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-28 20:53:37,979 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42693%2C1680036815956.meta, suffix=.meta, logDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/WALs/jenkins-hbase4.apache.org,42693,1680036815956, archiveDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/oldWALs, maxLogs=32 2023-03-28 20:53:37,995 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK] 2023-03-28 20:53:37,999 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/WALs/jenkins-hbase4.apache.org,42693,1680036815956/jenkins-hbase4.apache.org%2C42693%2C1680036815956.meta.1680036817980.meta 2023-03-28 20:53:37,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36815,DS-4916f720-d31e-4004-bdf3-8c8f70bf3833,DISK]] 2023-03-28 20:53:38,000 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:53:38,002 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-28 20:53:38,019 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-28 20:53:38,025 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-28 20:53:38,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-28 20:53:38,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:38,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-28 20:53:38,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-28 20:53:38,033 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-28 20:53:38,035 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/info 2023-03-28 20:53:38,035 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/info 2023-03-28 20:53:38,035 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-28 20:53:38,036 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:38,036 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-28 20:53:38,038 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/rep_barrier 2023-03-28 20:53:38,038 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/rep_barrier 2023-03-28 20:53:38,039 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-28 20:53:38,039 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:38,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-28 20:53:38,041 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/table 2023-03-28 20:53:38,041 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/table 2023-03-28 20:53:38,042 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-28 20:53:38,043 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:38,045 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740 2023-03-28 20:53:38,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740 2023-03-28 20:53:38,051 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:53:38,053 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-28 20:53:38,054 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63253651, jitterRate=-0.057447150349617004}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:53:38,055 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-28 20:53:38,064 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680036817950 2023-03-28 20:53:38,080 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-28 20:53:38,081 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-28 20:53:38,081 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42693,1680036815956, state=OPEN 2023-03-28 20:53:38,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-28 20:53:38,084 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-28 20:53:38,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-28 20:53:38,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42693,1680036815956 in 311 msec 2023-03-28 20:53:38,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-28 20:53:38,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 506 msec 2023-03-28 20:53:38,098 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 740 msec 2023-03-28 20:53:38,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680036818098, completionTime=-1 2023-03-28 20:53:38,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-28 20:53:38,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-28 20:53:38,160 DEBUG [hconnection-0x191d92e5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:38,164 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38708, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:38,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-28 20:53:38,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680036878182 2023-03-28 20:53:38,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680036938182 2023-03-28 20:53:38,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 83 msec 2023-03-28 20:53:38,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44521,1680036814691-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:38,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44521,1680036814691-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:38,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44521,1680036814691-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:38,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44521, period=300000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:38,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-28 20:53:38,213 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-28 20:53:38,221 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-28 20:53:38,223 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-28 20:53:38,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-28 20:53:38,234 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:53:38,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:53:38,258 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:38,260 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36 empty. 2023-03-28 20:53:38,261 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:38,261 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-28 20:53:38,682 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-28 20:53:38,684 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 097dd7e52a34fa343db86f0dfd060f36, NAME => 'hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:53:39,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:39,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 097dd7e52a34fa343db86f0dfd060f36, disabling compactions & flushes 2023-03-28 20:53:39,097 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. after waiting 0 ms 2023-03-28 20:53:39,097 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,098 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,098 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 097dd7e52a34fa343db86f0dfd060f36: 2023-03-28 20:53:39,103 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:53:39,119 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680036819107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036819107"}]},"ts":"1680036819107"} 2023-03-28 20:53:39,148 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:53:39,151 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:53:39,156 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036819151"}]},"ts":"1680036819151"} 2023-03-28 20:53:39,161 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-28 20:53:39,170 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=097dd7e52a34fa343db86f0dfd060f36, ASSIGN}] 2023-03-28 20:53:39,173 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=097dd7e52a34fa343db86f0dfd060f36, ASSIGN 2023-03-28 20:53:39,174 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=097dd7e52a34fa343db86f0dfd060f36, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:53:39,326 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=097dd7e52a34fa343db86f0dfd060f36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:39,326 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680036819325"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036819325"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036819325"}]},"ts":"1680036819325"} 2023-03-28 20:53:39,331 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 097dd7e52a34fa343db86f0dfd060f36, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:53:39,490 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,491 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 097dd7e52a34fa343db86f0dfd060f36, NAME => 'hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:53:39,492 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,492 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:39,492 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,492 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,494 INFO [StoreOpener-097dd7e52a34fa343db86f0dfd060f36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,496 DEBUG [StoreOpener-097dd7e52a34fa343db86f0dfd060f36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/info 2023-03-28 20:53:39,496 DEBUG [StoreOpener-097dd7e52a34fa343db86f0dfd060f36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/info 2023-03-28 20:53:39,497 INFO [StoreOpener-097dd7e52a34fa343db86f0dfd060f36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 097dd7e52a34fa343db86f0dfd060f36 columnFamilyName info 2023-03-28 20:53:39,497 INFO [StoreOpener-097dd7e52a34fa343db86f0dfd060f36-1] regionserver.HStore(310): Store=097dd7e52a34fa343db86f0dfd060f36/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:39,499 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,500 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,505 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:53:39,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:53:39,509 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 097dd7e52a34fa343db86f0dfd060f36; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73124261, jitterRate=0.08963640034198761}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-28 20:53:39,509 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 097dd7e52a34fa343db86f0dfd060f36: 2023-03-28 20:53:39,511 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36., pid=6, masterSystemTime=1680036819484 2023-03-28 20:53:39,515 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,515 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:53:39,517 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=097dd7e52a34fa343db86f0dfd060f36, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:39,517 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680036819516"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036819516"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036819516"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036819516"}]},"ts":"1680036819516"} 2023-03-28 20:53:39,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-28 20:53:39,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 097dd7e52a34fa343db86f0dfd060f36, server=jenkins-hbase4.apache.org,42693,1680036815956 in 190 msec 2023-03-28 20:53:39,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-28 20:53:39,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=097dd7e52a34fa343db86f0dfd060f36, ASSIGN in 354 msec 2023-03-28 20:53:39,528 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:53:39,529 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036819529"}]},"ts":"1680036819529"} 2023-03-28 20:53:39,532 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-28 20:53:39,536 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:53:39,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-28 20:53:39,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-28 20:53:39,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:53:39,539 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3130 sec 2023-03-28 20:53:39,575 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-28 20:53:39,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-28 20:53:39,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-03-28 20:53:39,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-28 20:53:39,620 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-28 20:53:39,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-28 20:53:39,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-28 20:53:39,640 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-28 20:53:39,640 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.603sec 2023-03-28 20:53:39,642 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-28 20:53:39,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-28 20:53:39,644 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-28 20:53:39,645 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44521,1680036814691-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-28 20:53:39,645 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44521,1680036814691-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-28 20:53:39,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-28 20:53:39,708 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x2d205425 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:39,713 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c01f436, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:39,726 DEBUG [hconnection-0x1d3af235-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:39,737 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38716, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:39,747 INFO [Listener at localhost/34739] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:53:39,767 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=416, ProcessCount=170, AvailableMemoryMB=7535 2023-03-28 20:53:39,782 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:53:39,785 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:53:39,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:53:39,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:53:39,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:53:39,802 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:53:39,804 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:53:39,807 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:39,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-28 20:53:39,808 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef empty. 2023-03-28 20:53:39,809 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:39,810 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:53:39,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:39,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:40,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:40,229 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:53:40,231 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 59dd79967ef9e13a346f2f7f11b173ef, NAME => 'TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:53:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:40,644 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:40,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 59dd79967ef9e13a346f2f7f11b173ef, disabling compactions & flushes 2023-03-28 20:53:40,645 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. after waiting 0 ms 2023-03-28 20:53:40,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,645 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:40,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:53:40,651 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036820651"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036820651"}]},"ts":"1680036820651"} 2023-03-28 20:53:40,654 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:53:40,656 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:53:40,656 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036820656"}]},"ts":"1680036820656"} 2023-03-28 20:53:40,658 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:53:40,663 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, ASSIGN}] 2023-03-28 20:53:40,665 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, ASSIGN 2023-03-28 20:53:40,666 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:53:40,817 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=59dd79967ef9e13a346f2f7f11b173ef, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:40,818 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036820817"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036820817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036820817"}]},"ts":"1680036820817"} 2023-03-28 20:53:40,822 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:53:40,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:40,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 59dd79967ef9e13a346f2f7f11b173ef, NAME => 'TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:53:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:53:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,983 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,985 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A 2023-03-28 20:53:40,985 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A 2023-03-28 20:53:40,987 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:53:40,987 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59dd79967ef9e13a346f2f7f11b173ef columnFamilyName A 2023-03-28 20:53:40,988 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(310): Store=59dd79967ef9e13a346f2f7f11b173ef/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:40,988 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,990 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B 2023-03-28 20:53:40,990 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B 2023-03-28 20:53:40,990 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:53:40,990 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59dd79967ef9e13a346f2f7f11b173ef columnFamilyName B 2023-03-28 20:53:40,991 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(310): Store=59dd79967ef9e13a346f2f7f11b173ef/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:40,991 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,993 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C 2023-03-28 20:53:40,993 DEBUG [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C 2023-03-28 20:53:40,993 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:53:40,993 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59dd79967ef9e13a346f2f7f11b173ef columnFamilyName C 2023-03-28 20:53:40,994 INFO [StoreOpener-59dd79967ef9e13a346f2f7f11b173ef-1] regionserver.HStore(310): Store=59dd79967ef9e13a346f2f7f11b173ef/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:53:40,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:40,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:40,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:41,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:53:41,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:41,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:53:41,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 59dd79967ef9e13a346f2f7f11b173ef; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65007142, jitterRate=-0.031318098306655884}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:53:41,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:41,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., pid=11, masterSystemTime=1680036820975 2023-03-28 20:53:41,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:41,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:41,010 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=59dd79967ef9e13a346f2f7f11b173ef, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:41,010 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036821010"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036821010"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036821010"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036821010"}]},"ts":"1680036821010"} 2023-03-28 20:53:41,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-28 20:53:41,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 in 192 msec 2023-03-28 20:53:41,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-28 20:53:41,020 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, ASSIGN in 353 msec 2023-03-28 20:53:41,021 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:53:41,021 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036821021"}]},"ts":"1680036821021"} 2023-03-28 20:53:41,023 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:53:41,027 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:53:41,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2290 sec 2023-03-28 20:53:41,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-28 20:53:41,935 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-28 20:53:41,940 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x36d9637e to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,946 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@348b992a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,949 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:41,952 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38720, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:41,955 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:53:41,958 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35778, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:53:41,966 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x6b41e306 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,970 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2732b8c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,970 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x2411a862 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,974 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72ea4a65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,975 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x377933e4 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,979 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1645b6b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,979 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x3e8196e0 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,984 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b52aa42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,984 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1ca4a062 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,988 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1710407d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,989 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x75f5a5e6 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:41,993 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ef1151, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:41,993 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x71f9f426 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:42,024 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29fba56a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:42,024 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x5ae6c42b to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:42,029 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ec2fae8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:42,029 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x726fb449 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:53:42,033 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e44f1b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:53:42,037 DEBUG [hconnection-0x29b3bfb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,040 DEBUG [hconnection-0x2b0f04f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,040 DEBUG [hconnection-0x4d06a665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,041 DEBUG [hconnection-0x27e1645f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,044 DEBUG [hconnection-0x5dfec272-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,045 DEBUG [hconnection-0x742302d5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,045 DEBUG [hconnection-0x6fed7337-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,045 DEBUG [hconnection-0x637b85c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,052 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,052 DEBUG [hconnection-0x66fde656-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:53:42,053 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38742, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,055 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38756, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,055 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38732, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,056 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38786, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:42,058 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38752, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,058 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,066 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,066 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:53:42,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:42,082 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:42,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:42,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:42,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:42,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:42,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:42,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:42,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:42,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:42,092 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:42,093 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:42,125 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-28 20:53:42,143 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:42,144 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:42,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:42,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:42,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:42,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:42,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:42,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:42,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036822208, max timestamp=1680036822249Num uniques 859; 2023-03-28 20:53:42,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:53:42,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036822208, max timestamp=1680036822251Num uniques 859; 2023-03-28 20:53:42,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-28 20:53:42,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036822208, max timestamp=1680036822239Num uniques 859; 2023-03-28 20:53:42,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-28 20:53:42,300 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/230c2c7b60f94544968671d2f8519a65 2023-03-28 20:53:42,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036882351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036822321, max timestamp=1680036822352Num uniques 1009; 2023-03-28 20:53:42,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-28 20:53:42,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036822321, max timestamp=1680036822352Num uniques 1018; 2023-03-28 20:53:42,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46616, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-28 20:53:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036822321, max timestamp=1680036822351Num uniques 1009; 2023-03-28 20:53:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-28 20:53:42,462 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/2ac3946f07d04ffa90f49ad3ff686a88 2023-03-28 20:53:42,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036822394, max timestamp=1680036822460Num uniques 1009; 2023-03-28 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46216, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-03-28 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036822394, max timestamp=1680036822460Num uniques 1009; 2023-03-28 20:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-03-28 20:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036822394, max timestamp=1680036822458Num uniques 1027; 2023-03-28 20:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-28 20:53:42,525 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/d15ee1c054504de0b1edb7f63cd66be5 2023-03-28 20:53:42,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680036822505, max timestamp=1680036822525Num uniques 1086; 2023-03-28 20:53:42,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-28 20:53:42,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680036822505, max timestamp=1680036822525Num uniques 1086; 2023-03-28 20:53:42,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-28 20:53:42,541 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/230c2c7b60f94544968671d2f8519a65 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65 2023-03-28 20:53:42,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036822505, max timestamp=1680036822524Num uniques 1086; 2023-03-28 20:53:42,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-03-28 20:53:42,561 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65, entries=100, sequenceid=8, filesize=9.3 K 2023-03-28 20:53:42,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/2ac3946f07d04ffa90f49ad3ff686a88 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88 2023-03-28 20:53:42,586 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88, entries=100, sequenceid=8, filesize=9.3 K 2023-03-28 20:53:42,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/d15ee1c054504de0b1edb7f63cd66be5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5 2023-03-28 20:53:42,608 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5, entries=100, sequenceid=8, filesize=9.3 K 2023-03-28 20:53:42,610 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=105.74 KB/108274 for 59dd79967ef9e13a346f2f7f11b173ef in 516ms, sequenceid=8, compaction requested=false 2023-03-28 20:53:42,613 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-28 20:53:42,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:42,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:42,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:42,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:42,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:42,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:42,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1152b627[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:42,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,631 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:42,631 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:42,631 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:42,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:42,635 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:42,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,635 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,635 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:42,641 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:42,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:42,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:42,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:42,641 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:42,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=112.45 KB heapSize=319.03 KB 2023-03-28 20:53:42,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:42,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:42,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:42,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:42,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:42,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:42,646 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,646 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:42,646 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:42,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:42,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:42,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:42,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:42,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036882708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,744 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,747 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,747 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:42,747 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:42,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036882721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036882719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036882737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036882738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=36.41 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/be191d5da85340f9bd5529c10a9f1313 2023-03-28 20:53:42,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/e86c45bf9c51429099cd75894ae0d652 2023-03-28 20:53:42,847 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:42,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:42,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:42,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036882849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036882858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036882858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036882859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:42,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036882859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:42,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:42,884 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:42,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:42,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:42,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:42,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,889 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:42,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:42,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:42,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:42,899 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:42,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:42,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:42,905 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:42,905 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:42,906 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:42,906 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:42,910 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:42,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d49951e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:42,914 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:42,915 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:42,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:42,925 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:42,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:42,931 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:42,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:42,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:42,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:42,931 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:42,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:42,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:42,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:42,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:42,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:42,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:42,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:42,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:42,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:42,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:42,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:42,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:42,960 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:43,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:43,071 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:43,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:43,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:43,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036883085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036883085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:43,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036883095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036883108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036883128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:43,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:43,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:43,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:43,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:43,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,155 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,155 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1e08b7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:43,158 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:43,158 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,161 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:43,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:43,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:43,168 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:43,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:43,171 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:43,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:43,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:43,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:43,172 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:43,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:43,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:43,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:43,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:43,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:43,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:43,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:43,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:43,307 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:43,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:43,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/863472eafeea41e8ab6278c76f6a2250 2023-03-28 20:53:43,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,323 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,323 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:43,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/be191d5da85340f9bd5529c10a9f1313 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313 2023-03-28 20:53:43,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:43,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:43,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:43,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:43,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:43,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:43,354 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:43,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22ba535[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:43,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:43,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313, entries=200, sequenceid=93, filesize=13.9 K 2023-03-28 20:53:43,360 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:43,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:43,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/e86c45bf9c51429099cd75894ae0d652 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652 2023-03-28 20:53:43,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:43,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:43,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:43,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:43,376 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:43,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:43,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:43,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:43,376 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:43,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652, entries=150, sequenceid=93, filesize=11.6 K 2023-03-28 20:53:43,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/863472eafeea41e8ab6278c76f6a2250 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250 2023-03-28 20:53:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,404 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,405 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,405 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,405 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,405 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,405 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:43,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:43,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:43,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036883403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:43,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036883412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250, entries=150, sequenceid=93, filesize=11.6 K 2023-03-28 20:53:43,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~112.45 KB/115144, heapSize ~318.98 KB/326640, currentSize=67.09 KB/68700 for 59dd79967ef9e13a346f2f7f11b173ef in 771ms, sequenceid=93, compaction requested=false 2023-03-28 20:53:43,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:43,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:43,428 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:43,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:43,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036883505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036883505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036883507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:43,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:43,524 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:43,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:43,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:43,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,527 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,527 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:43,530 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,530 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:43,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:43,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:43,535 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:43,535 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:43,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:43,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:43,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:43,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:43,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:43,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:43,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:43,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:43,544 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:43,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cc8d9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:43,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:43,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,550 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:43,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:43,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:43,563 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:43,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:43,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:43,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:43,563 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:43,567 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,567 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,567 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,578 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:43,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:43,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,581 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:43,581 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:43,620 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:53:43,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036883617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036883619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036883620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:43,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:43,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:43,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:43,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:43,693 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:43,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:43,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:43,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:43,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:43,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:43,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:43,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,715 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-28 20:53:43,718 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-28 20:53:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:43,722 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,722 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:43,722 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:43,723 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:43,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,726 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:43,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ac5c6b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:43,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:43,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,732 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:43,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:43,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:43,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:43,744 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:43,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:43,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:43,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:43,745 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:43,751 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,751 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,751 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,752 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:43,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:43,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:43,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036883826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036883827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036883830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:43,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:43,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:43,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:43,890 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:43,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:43,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:43,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:43,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:43,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036883914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:43,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/10d1925c786b472b95c91b8d264f2986 2023-03-28 20:53:43,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:43,925 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,925 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:43,925 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:43,925 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:43,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:43,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036883917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,934 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:43,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d6da799[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:43,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:43,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:43,945 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:43,945 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:43,946 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:43,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:43,953 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:43,953 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:43,953 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:43,953 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:43,953 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:43,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:43,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:43,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:43,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,960 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,964 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:43,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:43,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:43,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:43,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:43,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:43,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:43,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:43,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:43,967 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:43,967 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:43,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:43,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:43,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:43,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/8d9165fa6e7248fa838cc2dde6d6f4a3 2023-03-28 20:53:44,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:44,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:44,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:44,078 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:44,078 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:44,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:44,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:44,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:44,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:44,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:44,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:44,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:44,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:44,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:44,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:44,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:44,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:44,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4230ab0c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:44,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:44,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:44,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:44,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:44,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:44,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:44,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:44,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:44,103 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:44,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:44,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:44,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:44,103 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:44,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:44,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,120 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:44,125 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:44,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:44,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036884135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036884139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036884137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,225 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:44,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:44,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:44,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:44,253 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:44,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:44,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:44,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:44,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:44,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:44,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:44,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:44,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:44,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:44,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:44,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:44,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,269 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22daa993[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:44,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:44,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,277 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:44,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:44,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:44,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:44,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:44,286 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:44,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:44,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:44,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:44,286 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:44,288 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,289 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,290 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:44,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,297 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:44,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:44,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:44,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:44,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:44,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:44,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:44,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:44,405 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:44,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:44,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:44,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,411 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:44,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:44,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:44,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:44,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:44,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:44,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:44,421 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4770f6a7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:44,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:44,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,428 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:44,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:44,432 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:44,440 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:44,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:44,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:44,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:44,440 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:44,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:44,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:44,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:44,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:44,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:44,451 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:44,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:44,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:44,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:44,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:44,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/df202e1913c34c1db53c9642241b18a6 2023-03-28 20:53:44,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/10d1925c786b472b95c91b8d264f2986 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986 2023-03-28 20:53:44,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986, entries=200, sequenceid=107, filesize=13.9 K 2023-03-28 20:53:44,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/8d9165fa6e7248fa838cc2dde6d6f4a3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3 2023-03-28 20:53:44,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3, entries=150, sequenceid=107, filesize=11.6 K 2023-03-28 20:53:44,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/df202e1913c34c1db53c9642241b18a6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6 2023-03-28 20:53:44,551 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:44,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:44,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:44,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:44,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:44,560 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:44,560 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:44,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:44,562 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,562 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,564 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:44,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:44,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:44,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6, entries=150, sequenceid=107, filesize=11.6 K 2023-03-28 20:53:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:44,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:44,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 59dd79967ef9e13a346f2f7f11b173ef in 1142ms, sequenceid=107, compaction requested=true 2023-03-28 20:53:44,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:44,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:44,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:44,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:44,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:44,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:44,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:44,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:44,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:44,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-28 20:53:44,587 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:44,587 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:44,588 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,588 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:44,588 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:53:44,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:44,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:44,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:44,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:44,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:44,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:44,594 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:44,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37994 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:44,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:44,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:44,596 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,596 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,597 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=32.5 K 2023-03-28 20:53:44,597 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=37.1 K 2023-03-28 20:53:44,599 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2ac3946f07d04ffa90f49ad3ff686a88, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680036822081 2023-03-28 20:53:44,603 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 230c2c7b60f94544968671d2f8519a65, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680036822081 2023-03-28 20:53:44,604 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e86c45bf9c51429099cd75894ae0d652, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680036822550 2023-03-28 20:53:44,605 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8d9165fa6e7248fa838cc2dde6d6f4a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:44,605 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting be191d5da85340f9bd5529c10a9f1313, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680036822550 2023-03-28 20:53:44,607 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 10d1925c786b472b95c91b8d264f2986, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:44,648 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#10 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:44,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:44,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:44,681 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#11 average throughput is 0.20 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:44,763 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/7da748b834044aa4a1555d183f888f76 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7da748b834044aa4a1555d183f888f76 2023-03-28 20:53:44,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036884760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036884762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,779 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/65514e2b7d9340768c9ab0d7eca00608 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/65514e2b7d9340768c9ab0d7eca00608 2023-03-28 20:53:44,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036884769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,792 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 7da748b834044aa4a1555d183f888f76(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:44,792 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:44,792 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036824585; duration=0sec 2023-03-28 20:53:44,793 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:44,793 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:44,802 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:44,802 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:44,803 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:44,803 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=32.5 K 2023-03-28 20:53:44,805 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d15ee1c054504de0b1edb7f63cd66be5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680036822081 2023-03-28 20:53:44,806 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 863472eafeea41e8ab6278c76f6a2250, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680036822550 2023-03-28 20:53:44,807 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting df202e1913c34c1db53c9642241b18a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:44,809 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 65514e2b7d9340768c9ab0d7eca00608(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:44,812 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:44,812 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036824572; duration=0sec 2023-03-28 20:53:44,812 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:44,831 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#12 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:44,862 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-28 20:53:44,863 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-28 20:53:44,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036884876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036884876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036884883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036884923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:44,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:44,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036884933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036885083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036885084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036885088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,095 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/7f81a7b6e7f648999f375b4c00ef1ffa 2023-03-28 20:53:45,275 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/fc411034a6c74af18c8ca8998a82b56b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/fc411034a6c74af18c8ca8998a82b56b 2023-03-28 20:53:45,287 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into fc411034a6c74af18c8ca8998a82b56b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:45,288 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:45,288 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=13, startTime=1680036824587; duration=0sec 2023-03-28 20:53:45,289 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:45,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036885390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036885391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036885391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,539 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/a5e48a20b3924cd1be98f0999aa4f20a 2023-03-28 20:53:45,586 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ff09de3e47324036829a172ed758a29b 2023-03-28 20:53:45,596 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/7f81a7b6e7f648999f375b4c00ef1ffa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa 2023-03-28 20:53:45,605 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa, entries=150, sequenceid=125, filesize=11.6 K 2023-03-28 20:53:45,608 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/a5e48a20b3924cd1be98f0999aa4f20a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a 2023-03-28 20:53:45,620 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a, entries=150, sequenceid=125, filesize=11.6 K 2023-03-28 20:53:45,631 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ff09de3e47324036829a172ed758a29b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b 2023-03-28 20:53:45,643 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b, entries=150, sequenceid=125, filesize=11.6 K 2023-03-28 20:53:45,649 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 59dd79967ef9e13a346f2f7f11b173ef in 1061ms, sequenceid=125, compaction requested=false 2023-03-28 20:53:45,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:45,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:45,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:45,654 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3751c0bf[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:45,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:45,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:45,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:45,662 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:45,685 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:45,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:45,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:45,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:45,685 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:45,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:45,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:45,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:45,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:45,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,703 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:45,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:45,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:45,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:45,704 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:45,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:45,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:45,812 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:45,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:45,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:45,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:45,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:45,818 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,818 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:45,818 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,818 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:45,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:45,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:45,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:45,821 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:45,822 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:45,822 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:45,822 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:53:45,822 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:45,848 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/d243d43439964ad29861840bdfd73bac 2023-03-28 20:53:45,879 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/bd7bd90c8ae34784ad99b49c35c5fa1b 2023-03-28 20:53:45,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:45,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:45,909 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1cde56e12380466b99fa27c63cf3abd7 2023-03-28 20:53:45,923 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/d243d43439964ad29861840bdfd73bac as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac 2023-03-28 20:53:45,934 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:53:45,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036885930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036885933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,940 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/bd7bd90c8ae34784ad99b49c35c5fa1b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b 2023-03-28 20:53:45,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:45,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036885937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,949 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:53:45,962 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1cde56e12380466b99fa27c63cf3abd7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7 2023-03-28 20:53:45,973 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:53:45,977 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 59dd79967ef9e13a346f2f7f11b173ef in 155ms, sequenceid=142, compaction requested=true 2023-03-28 20:53:45,977 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:45,977 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:45,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe6592[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:45,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:45,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:45,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:45,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:46,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:46,001 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:46,001 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:46,001 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:46,001 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:46,001 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:46,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:46,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:46,021 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:46,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:46,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:46,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036886076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036886077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036886078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:46,126 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:46,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,127 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:46,127 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:46,128 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,128 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:46,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:46,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:46,131 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:46,131 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:46,133 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,133 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:46,133 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:46,133 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,135 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:46,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71fdc2f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:46,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:46,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:46,139 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,139 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:46,142 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:46,145 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:46,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:46,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:46,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:46,146 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:46,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,156 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,156 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,156 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:46,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:46,156 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,157 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:46,157 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:46,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036886181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036886185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036886186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:46,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:46,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:46,265 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:46,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:46,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:46,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:46,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:46,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:46,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:46,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:46,277 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,278 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:46,278 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:46,278 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:46,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471ac7a5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:46,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:46,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:46,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:46,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:46,292 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:46,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:46,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:46,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:46,292 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:46,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,297 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,299 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:46,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:46,309 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:46,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:46,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036886385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036886388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036886389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:46,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:46,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:46,417 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:46,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:46,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:46,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,422 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:46,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:46,429 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,429 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:46,431 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:46,431 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:46,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef7a3c1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:46,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:46,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,449 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:46,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:46,453 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:46,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:46,466 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:46,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:46,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:46,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:46,466 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:46,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/138331c7117147efac3a7dd1d05e469d 2023-03-28 20:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:46,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:46,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:46,478 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:46,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:46,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:46,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:46,483 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:46,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/f7e73b2e424f4bdcb41d3e2076f96a37 2023-03-28 20:53:46,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/f660a7f9672c473c99581fd51635aa9d 2023-03-28 20:53:46,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/138331c7117147efac3a7dd1d05e469d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d 2023-03-28 20:53:46,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d, entries=250, sequenceid=162, filesize=16.4 K 2023-03-28 20:53:46,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/f7e73b2e424f4bdcb41d3e2076f96a37 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37 2023-03-28 20:53:46,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37, entries=150, sequenceid=162, filesize=11.8 K 2023-03-28 20:53:46,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/f660a7f9672c473c99581fd51635aa9d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d 2023-03-28 20:53:46,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:46,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:46,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:46,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:46,596 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:46,596 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:46,599 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:46,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d, entries=150, sequenceid=162, filesize=11.8 K 2023-03-28 20:53:46,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 59dd79967ef9e13a346f2f7f11b173ef in 560ms, sequenceid=162, compaction requested=true 2023-03-28 20:53:46,605 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:46,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:46,605 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:46,606 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:46,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:46,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:46,607 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,607 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:46,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:46,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:46,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:46,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:46,615 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:46,616 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:53:46,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:46,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,616 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:46,618 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47947 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:46,618 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:46,618 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,618 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7da748b834044aa4a1555d183f888f76, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=46.8 K 2023-03-28 20:53:46,619 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7da748b834044aa4a1555d183f888f76, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:46,620 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52727 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:46,621 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:46,621 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:46,621 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/65514e2b7d9340768c9ab0d7eca00608, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=51.5 K 2023-03-28 20:53:46,626 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a5e48a20b3924cd1be98f0999aa4f20a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680036823485 2023-03-28 20:53:46,627 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 65514e2b7d9340768c9ab0d7eca00608, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:46,628 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bd7bd90c8ae34784ad99b49c35c5fa1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036824659 2023-03-28 20:53:46,629 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7f81a7b6e7f648999f375b4c00ef1ffa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680036823485 2023-03-28 20:53:46,631 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d243d43439964ad29861840bdfd73bac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036824659 2023-03-28 20:53:46,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 138331c7117147efac3a7dd1d05e469d, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:46,634 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f7e73b2e424f4bdcb41d3e2076f96a37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:46,662 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/0cb28a788bf0441cba3e72aa830f749a 2023-03-28 20:53:46,679 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#22 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:46,687 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#23 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:46,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:46,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:46,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036886749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036886749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036886750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036886856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036886857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036886860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036886941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,943 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4222 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:46,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:46,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036886952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:46,958 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4218 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036887064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036887070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036887071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,123 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/fb9d4079b84d4aa58ce791cd6b2b02ca 2023-03-28 20:53:47,131 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/7af652d283654d1b9aa3a3196ba75c8c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7af652d283654d1b9aa3a3196ba75c8c 2023-03-28 20:53:47,138 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/b35dca0f42c64defa8b9a30dbb3a2f0b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/b35dca0f42c64defa8b9a30dbb3a2f0b 2023-03-28 20:53:47,149 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into b35dca0f42c64defa8b9a30dbb3a2f0b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:47,149 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:47,149 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=12, startTime=1680036826605; duration=0sec 2023-03-28 20:53:47,149 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:47,149 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:47,156 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47947 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:47,157 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:47,157 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:47,157 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/fc411034a6c74af18c8ca8998a82b56b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=46.8 K 2023-03-28 20:53:47,158 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fc411034a6c74af18c8ca8998a82b56b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680036822702 2023-03-28 20:53:47,158 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ff09de3e47324036829a172ed758a29b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680036823485 2023-03-28 20:53:47,159 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1cde56e12380466b99fa27c63cf3abd7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036824659 2023-03-28 20:53:47,161 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 7af652d283654d1b9aa3a3196ba75c8c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:47,180 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:47,180 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=12, startTime=1680036826605; duration=0sec 2023-03-28 20:53:47,180 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:47,161 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f660a7f9672c473c99581fd51635aa9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:47,205 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#26 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:47,233 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/b3857160bf67499597d63f113573e2cd as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b3857160bf67499597d63f113573e2cd 2023-03-28 20:53:47,242 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into b3857160bf67499597d63f113573e2cd(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:47,242 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:47,242 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=12, startTime=1680036826606; duration=0sec 2023-03-28 20:53:47,242 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:47,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036887372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036887373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036887376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,598 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/0a4f54a237ca471cbdf43fd6be9c58ed 2023-03-28 20:53:47,612 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/0cb28a788bf0441cba3e72aa830f749a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a 2023-03-28 20:53:47,622 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a, entries=150, sequenceid=174, filesize=11.8 K 2023-03-28 20:53:47,625 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/fb9d4079b84d4aa58ce791cd6b2b02ca as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca 2023-03-28 20:53:47,634 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca, entries=150, sequenceid=174, filesize=11.8 K 2023-03-28 20:53:47,636 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/0a4f54a237ca471cbdf43fd6be9c58ed as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed 2023-03-28 20:53:47,643 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed, entries=150, sequenceid=174, filesize=11.8 K 2023-03-28 20:53:47,644 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 59dd79967ef9e13a346f2f7f11b173ef in 1028ms, sequenceid=174, compaction requested=false 2023-03-28 20:53:47,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:47,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,650 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d269b78[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:47,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:47,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,658 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:47,658 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,658 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:47,660 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:47,667 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:47,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:47,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:47,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:47,667 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:47,670 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,670 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:47,672 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:47,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:47,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:47,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:47,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:47,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:47,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:47,699 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:47,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:47,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:47,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:47,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:47,822 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:47,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:47,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:47,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:47,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:47,825 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:47,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:47,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:47,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:47,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:47,830 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:47,830 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:47,830 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:53:47,830 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:47,831 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:47,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:47,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:47,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:47,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:47,873 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/0942472af10146959169141f2c2df945 2023-03-28 20:53:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:47,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:47,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036887901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036887909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036887909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:47,928 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/147cbcb148b7464982cea2f8fa2dcec6 2023-03-28 20:53:48,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036888011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036888015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036888013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036888215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036888224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036888224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,365 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/91591e4ba57b4e7597275b437631cd52 2023-03-28 20:53:48,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/0942472af10146959169141f2c2df945 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945 2023-03-28 20:53:48,383 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:53:48,386 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/147cbcb148b7464982cea2f8fa2dcec6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6 2023-03-28 20:53:48,394 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:53:48,397 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/91591e4ba57b4e7597275b437631cd52 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52 2023-03-28 20:53:48,407 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:53:48,415 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 59dd79967ef9e13a346f2f7f11b173ef in 583ms, sequenceid=197, compaction requested=true 2023-03-28 20:53:48,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:48,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:48,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1de57[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:48,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:48,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,423 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:48,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:48,427 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:48,432 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:48,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:48,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:48,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:48,432 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:48,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,435 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:48,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:48,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,440 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,446 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:48,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:48,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:48,525 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:53:48,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:48,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:48,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:48,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:48,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:48,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:48,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/bf147ba6c72d491889cc86fc1aa4539f 2023-03-28 20:53:48,546 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:48,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:48,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:48,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:48,554 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:48,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:48,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:48,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:48,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:48,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:48,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:48,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:48,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:48,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:48,576 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15170689[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:48,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:48,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,579 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:48,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:48,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:48,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:48,584 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:48,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:48,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:48,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:48,584 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:48,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,593 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036888593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:48,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:48,598 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:48,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:48,598 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,600 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,600 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036888595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036888598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:48,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:48,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:48,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036888699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036888704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036888704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:48,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:48,728 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:48,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,741 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:48,741 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:48,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,744 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:48,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:48,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:48,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:48,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:48,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd4641[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:48,768 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:48,768 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:48,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:48,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:48,785 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:48,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:48,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:48,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:48,785 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:48,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:48,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:48,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,797 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:48,797 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:48,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:48,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:48,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:48,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:48,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:48,903 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:48,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:48,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:48,906 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,906 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036888907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:48,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036888908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:48,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:48,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036888909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:48,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:48,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:48,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:48,915 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:48,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:48,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,922 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f89eab5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:48,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:48,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:48,934 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:48,955 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:48,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:48,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:48,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:48,955 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:48,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:48,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,961 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:48,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:48,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:48,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:48,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:48,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:48,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:48,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:48,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:48,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/4b0f17fc813041e7bfa87b31a30110f2 2023-03-28 20:53:48,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:48,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:48,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:48,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:48,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:48,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:49,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:49,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:49,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:49,083 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:49,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:49,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:49,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:49,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:49,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:49,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:49,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:49,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:49,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1550cc75[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:49,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:49,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,103 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:49,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:49,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:49,104 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,104 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:49,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:49,108 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:49,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:49,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:49,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:49,108 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:49,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,113 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:49,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:49,132 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:49,132 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,132 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036889213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036889215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036889216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,232 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:49,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:49,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:49,240 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:49,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:49,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:49,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:49,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:49,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:49,249 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,249 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:49,249 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:49,249 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:49,251 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38dea318[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:49,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:49,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:49,262 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:49,266 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:49,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:49,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:49,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:49,266 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:49,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,270 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:49,276 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:49,276 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:49,376 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:49,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:49,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:49,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:49,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:49,382 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:49,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:49,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:49,383 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,383 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:49,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:49,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,388 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:49,394 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:49,395 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:49,397 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:49,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed750c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:49,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:49,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:49,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:49,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:49,410 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:49,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:49,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:49,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:49,410 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:49,412 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,412 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,412 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:49,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:49,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:49,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,420 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:49,420 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:49,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/75b157a409ac4fff9a8a62e63b32875f 2023-03-28 20:53:49,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/bf147ba6c72d491889cc86fc1aa4539f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f 2023-03-28 20:53:49,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f, entries=200, sequenceid=210, filesize=14.1 K 2023-03-28 20:53:49,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/4b0f17fc813041e7bfa87b31a30110f2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2 2023-03-28 20:53:49,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2, entries=150, sequenceid=210, filesize=11.8 K 2023-03-28 20:53:49,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/75b157a409ac4fff9a8a62e63b32875f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f 2023-03-28 20:53:49,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f, entries=150, sequenceid=210, filesize=11.8 K 2023-03-28 20:53:49,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 59dd79967ef9e13a346f2f7f11b173ef in 952ms, sequenceid=210, compaction requested=true 2023-03-28 20:53:49,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:49,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:49,478 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:49,478 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:49,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:49,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:49,480 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:49,480 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:49,480 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50774 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:49,480 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,480 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:49,480 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/b35dca0f42c64defa8b9a30dbb3a2f0b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=47.3 K 2023-03-28 20:53:49,480 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,481 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7af652d283654d1b9aa3a3196ba75c8c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=49.6 K 2023-03-28 20:53:49,481 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b35dca0f42c64defa8b9a30dbb3a2f0b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:49,482 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7af652d283654d1b9aa3a3196ba75c8c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:49,484 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fb9d4079b84d4aa58ce791cd6b2b02ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680036826063 2023-03-28 20:53:49,485 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0cb28a788bf0441cba3e72aa830f749a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680036826063 2023-03-28 20:53:49,485 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 147cbcb148b7464982cea2f8fa2dcec6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036826733 2023-03-28 20:53:49,486 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0942472af10146959169141f2c2df945, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036826733 2023-03-28 20:53:49,486 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4b0f17fc813041e7bfa87b31a30110f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827892 2023-03-28 20:53:49,486 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting bf147ba6c72d491889cc86fc1aa4539f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827887 2023-03-28 20:53:49,517 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:49,520 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:49,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:49,517 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:49,545 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:49,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:49,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:49,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:49,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:49,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:49,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:49,562 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:53:49,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:49,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:49,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:49,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/496ba064aa674c46adff6b37b3892d6c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/496ba064aa674c46adff6b37b3892d6c 2023-03-28 20:53:49,612 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 496ba064aa674c46adff6b37b3892d6c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:49,612 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:49,612 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=12, startTime=1680036829478; duration=0sec 2023-03-28 20:53:49,612 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:49,612 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:53:49,632 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:53:49,632 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:49,632 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,633 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b3857160bf67499597d63f113573e2cd, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=47.3 K 2023-03-28 20:53:49,633 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b3857160bf67499597d63f113573e2cd, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036825923 2023-03-28 20:53:49,634 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0a4f54a237ca471cbdf43fd6be9c58ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680036826063 2023-03-28 20:53:49,638 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 91591e4ba57b4e7597275b437631cd52, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036826733 2023-03-28 20:53:49,639 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 75b157a409ac4fff9a8a62e63b32875f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827892 2023-03-28 20:53:49,647 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/5820c486cf8b41ceaa6f572eacd04756 2023-03-28 20:53:49,670 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:49,711 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/4dcfaf69dcc748a0b1f54873f17dcda5 2023-03-28 20:53:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:49,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036889745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036889746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036889747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,762 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/903518400b4e4892bdc13ff0ca2a6961 2023-03-28 20:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/5820c486cf8b41ceaa6f572eacd04756 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756 2023-03-28 20:53:49,776 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756, entries=150, sequenceid=229, filesize=11.8 K 2023-03-28 20:53:49,779 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/4dcfaf69dcc748a0b1f54873f17dcda5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5 2023-03-28 20:53:49,785 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5, entries=150, sequenceid=229, filesize=11.8 K 2023-03-28 20:53:49,786 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/903518400b4e4892bdc13ff0ca2a6961 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961 2023-03-28 20:53:49,801 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961, entries=150, sequenceid=229, filesize=11.8 K 2023-03-28 20:53:49,802 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 59dd79967ef9e13a346f2f7f11b173ef in 240ms, sequenceid=229, compaction requested=false 2023-03-28 20:53:49,802 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:49,802 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,802 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:49,804 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:49,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e1eb66[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:49,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:49,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:49,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:49,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:49,810 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:49,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:49,813 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:49,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:49,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:49,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:49,814 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:49,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,818 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,824 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,825 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,825 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:49,825 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:49,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:49,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:49,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/1d7db8cddd4247cd89c77a0f3c03ead0 2023-03-28 20:53:49,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036889876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036889881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036889883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,925 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:49,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:49,930 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:49,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:49,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:49,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,934 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,934 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:49,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:49,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:49,936 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,936 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:49,936 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:49,937 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:49,938 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4741a6eb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:49,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:49,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:49,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:49,943 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,951 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/fa49a995454d49a386c0ccb13675613a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fa49a995454d49a386c0ccb13675613a 2023-03-28 20:53:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:49,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:49,959 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:49,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:49,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:49,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:49,960 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:49,961 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into fa49a995454d49a386c0ccb13675613a(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:49,962 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:49,962 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=12, startTime=1680036829477; duration=0sec 2023-03-28 20:53:49,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:49,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,964 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:49,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:49,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:49,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:49,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:49,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:49,969 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:49,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:49,969 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:49,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036889984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036889987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:49,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:49,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036889988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,069 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:50,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:50,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:50,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:50,074 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:50,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:50,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:50,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,076 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,077 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,077 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:50,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:50,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:50,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:50,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:50,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:50,081 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aca0b78[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:50,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:50,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:50,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:50,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:50,088 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:50,092 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:50,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:50,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:50,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:50,093 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:50,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,106 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:50,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:50,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,106 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:50,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:50,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:50,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,121 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/aab71f48741d4691b3ce348b3363bb90 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aab71f48741d4691b3ce348b3363bb90 2023-03-28 20:53:50,130 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into aab71f48741d4691b3ce348b3363bb90(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:50,130 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,130 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=12, startTime=1680036829478; duration=0sec 2023-03-28 20:53:50,130 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036890189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036890192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38742 deadline: 1680036890196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,206 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:50,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:50,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:50,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:50,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:50,210 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:50,210 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:50,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:50,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,213 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,213 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,213 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:50,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:50,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:50,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:50,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:50,216 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,218 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e05a38[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:50,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:50,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,222 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:50,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:50,223 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:50,223 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:50,224 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:50,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:50,228 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:50,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:50,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:50,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:50,228 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:50,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:50,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:50,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,236 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:50,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,236 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:50,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/14698d37869d4fc88f5d1ced02261402 2023-03-28 20:53:50,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/83d54fd5b4da4b42966b80db65a49bb3 2023-03-28 20:53:50,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/1d7db8cddd4247cd89c77a0f3c03ead0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0 2023-03-28 20:53:50,336 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:50,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0, entries=150, sequenceid=246, filesize=11.8 K 2023-03-28 20:53:50,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:50,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/14698d37869d4fc88f5d1ced02261402 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402 2023-03-28 20:53:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:50,341 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:50,341 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,341 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:50,341 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:50,342 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,342 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,343 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:50,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:50,346 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,346 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:50,346 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:50,346 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:50,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402, entries=150, sequenceid=246, filesize=11.8 K 2023-03-28 20:53:50,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/83d54fd5b4da4b42966b80db65a49bb3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3 2023-03-28 20:53:50,353 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,353 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,353 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,353 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e813739[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:50,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:50,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:50,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:50,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:50,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:50,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3, entries=150, sequenceid=246, filesize=11.8 K 2023-03-28 20:53:50,361 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:50,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 59dd79967ef9e13a346f2f7f11b173ef in 510ms, sequenceid=246, compaction requested=true 2023-03-28 20:53:50,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:50,362 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:50,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:50,362 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,363 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:50,364 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:50,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:50,364 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:50,364 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fa49a995454d49a386c0ccb13675613a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=35.6 K 2023-03-28 20:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,364 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,369 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/496ba064aa674c46adff6b37b3892d6c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=35.6 K 2023-03-28 20:53:50,364 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fa49a995454d49a386c0ccb13675613a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827892 2023-03-28 20:53:50,370 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 496ba064aa674c46adff6b37b3892d6c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827892 2023-03-28 20:53:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:50,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:50,371 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:50,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:50,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:50,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:50,371 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:50,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5820c486cf8b41ceaa6f572eacd04756, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680036828589 2023-03-28 20:53:50,372 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4dcfaf69dcc748a0b1f54873f17dcda5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680036828589 2023-03-28 20:53:50,372 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1d7db8cddd4247cd89c77a0f3c03ead0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:50,372 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 14698d37869d4fc88f5d1ced02261402, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:50,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,383 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#42 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:50,384 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,384 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,384 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:50,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:50,384 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,384 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:50,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:50,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:50,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,387 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:50,407 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/f567d71c53ab4f3a93f4c5c4a04a4182 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f567d71c53ab4f3a93f4c5c4a04a4182 2023-03-28 20:53:50,416 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into f567d71c53ab4f3a93f4c5c4a04a4182(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:50,416 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,416 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036830362; duration=0sec 2023-03-28 20:53:50,416 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:50,416 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:50,417 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:50,417 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:50,417 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,417 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aab71f48741d4691b3ce348b3363bb90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=35.6 K 2023-03-28 20:53:50,418 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting aab71f48741d4691b3ce348b3363bb90, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680036827892 2023-03-28 20:53:50,419 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 903518400b4e4892bdc13ff0ca2a6961, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680036828589 2023-03-28 20:53:50,419 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 83d54fd5b4da4b42966b80db65a49bb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:50,428 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#44 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:50,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:50,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:50,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:50,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:50,495 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:50,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:50,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:50,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:50,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,498 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:50,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:50,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:50,504 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,504 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:50,504 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:50,504 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:50,520 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bda8fc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:50,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:50,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,525 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:50,526 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,526 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:50,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036890520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036890522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036890525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:50,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/d3fd354eea054a0599c434139ba74c94 2023-03-28 20:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:50,535 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:50,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:50,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:50,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:50,535 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:50,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:50,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:50,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:50,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:50,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:50,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:50,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:50,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:50,548 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:50,548 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:50,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/871a06864d2f448d9cce1211b46c4606 2023-03-28 20:53:50,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/dd948e74c24d4eb78658cc9aa743db21 2023-03-28 20:53:50,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/d3fd354eea054a0599c434139ba74c94 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94 2023-03-28 20:53:50,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38752 deadline: 1680036890628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38756 deadline: 1680036890628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94, entries=200, sequenceid=266, filesize=14.3 K 2023-03-28 20:53:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036890630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/871a06864d2f448d9cce1211b46c4606 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606 2023-03-28 20:53:50,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606, entries=150, sequenceid=266, filesize=11.9 K 2023-03-28 20:53:50,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/dd948e74c24d4eb78658cc9aa743db21 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21 2023-03-28 20:53:50,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:50,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:50,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:50,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21, entries=150, sequenceid=266, filesize=11.9 K 2023-03-28 20:53:50,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 59dd79967ef9e13a346f2f7f11b173ef in 158ms, sequenceid=266, compaction requested=false 2023-03-28 20:53:50,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:50,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:50,657 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:50,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:50,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:50,658 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,658 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:50,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:50,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:50,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:50,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:50,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:50,672 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:50,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:50,737 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/eb346804f8194fd4b59ecd1b8d9ea51a 2023-03-28 20:53:50,808 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/270cc39210d745269893cfa30851952c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/270cc39210d745269893cfa30851952c 2023-03-28 20:53:50,817 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 270cc39210d745269893cfa30851952c(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:50,818 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,818 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036830362; duration=0sec 2023-03-28 20:53:50,819 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:50,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:50,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:50,843 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/c51d222c0d2342fb95683d6fb85f2ac0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c51d222c0d2342fb95683d6fb85f2ac0 2023-03-28 20:53:50,849 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into c51d222c0d2342fb95683d6fb85f2ac0(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:50,849 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:50,849 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=13, startTime=1680036830362; duration=0sec 2023-03-28 20:53:50,850 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:50,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036890855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036890856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036890856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036890960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036890961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036890964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036890978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,983 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8262 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:50,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:50,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036890994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:50,997 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8257 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:53:51,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036891163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036891166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036891168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,195 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/c93b72e6515e44a7a6b400b4d06adb47 2023-03-28 20:53:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036891468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036891471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036891471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,619 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/5642388daf3c44d09bf46076d140a753 2023-03-28 20:53:51,632 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/eb346804f8194fd4b59ecd1b8d9ea51a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a 2023-03-28 20:53:51,640 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a, entries=150, sequenceid=281, filesize=11.9 K 2023-03-28 20:53:51,642 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/c93b72e6515e44a7a6b400b4d06adb47 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47 2023-03-28 20:53:51,650 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47, entries=150, sequenceid=281, filesize=11.9 K 2023-03-28 20:53:51,652 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/5642388daf3c44d09bf46076d140a753 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753 2023-03-28 20:53:51,665 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753, entries=150, sequenceid=281, filesize=11.9 K 2023-03-28 20:53:51,666 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 59dd79967ef9e13a346f2f7f11b173ef in 994ms, sequenceid=281, compaction requested=true 2023-03-28 20:53:51,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:51,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:51,680 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e76ecd8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:51,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:51,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:51,689 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:51,696 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:51,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:51,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:51,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:51,696 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:51,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:51,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:51,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:51,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:51,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,718 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:51,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:51,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:51,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:51,819 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:51,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:51,828 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:51,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:51,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:51,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:51,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:51,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,830 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:51,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:51,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:51,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:51,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:51,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:51,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:51,844 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:51,844 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:51,844 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:51,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:51,910 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/14cdd40ac5434eb08d2d58ccc4b37b21 2023-03-28 20:53:51,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:51,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:52,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,331 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/78e71b69bf7a4de9ab2ad8bbf6130e63 2023-03-28 20:53:52,359 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/67ca5113b76a4caf81cb49b23232ebe8 2023-03-28 20:53:52,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/14cdd40ac5434eb08d2d58ccc4b37b21 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21 2023-03-28 20:53:52,379 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21, entries=150, sequenceid=300, filesize=11.9 K 2023-03-28 20:53:52,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/78e71b69bf7a4de9ab2ad8bbf6130e63 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63 2023-03-28 20:53:52,386 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63, entries=150, sequenceid=300, filesize=11.9 K 2023-03-28 20:53:52,387 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/67ca5113b76a4caf81cb49b23232ebe8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8 2023-03-28 20:53:52,395 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8, entries=150, sequenceid=300, filesize=11.9 K 2023-03-28 20:53:52,397 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 59dd79967ef9e13a346f2f7f11b173ef in 553ms, sequenceid=300, compaction requested=true 2023-03-28 20:53:52,397 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:52,397 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:52,397 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:52,397 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:52,397 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:52,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492052b4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:52,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:52,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:52,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:52,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:52,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:52,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:52,411 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:52,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:52,415 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:52,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:52,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:52,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:52,415 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:52,417 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,417 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:52,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,417 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:52,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:52,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:52,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:52,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:52,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:52,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,426 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:52,426 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:52,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:52,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:52,527 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:52,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:52,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:52,531 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:52,531 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:52,531 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:52,531 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:52,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:52,533 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:52,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:52,534 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,534 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:52,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:52,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:52,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:52,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:52,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:52,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:52,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:52,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:52,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:52,538 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:52,538 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:52,539 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:52,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:52,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036892951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036892951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:52,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036892951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:52,955 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/dc207d577f4043b687b46dc653b4b1ff 2023-03-28 20:53:52,975 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/e96caf793bf541639343145171d1ce7e 2023-03-28 20:53:52,998 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ff1ddbc858764ece983af3327e4d5613 2023-03-28 20:53:53,007 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/dc207d577f4043b687b46dc653b4b1ff as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff 2023-03-28 20:53:53,013 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff, entries=150, sequenceid=315, filesize=11.9 K 2023-03-28 20:53:53,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/e96caf793bf541639343145171d1ce7e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e 2023-03-28 20:53:53,021 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e, entries=150, sequenceid=315, filesize=11.9 K 2023-03-28 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ff1ddbc858764ece983af3327e4d5613 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613 2023-03-28 20:53:53,028 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613, entries=150, sequenceid=315, filesize=11.9 K 2023-03-28 20:53:53,029 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 59dd79967ef9e13a346f2f7f11b173ef in 491ms, sequenceid=315, compaction requested=true 2023-03-28 20:53:53,029 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:53,029 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:53,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eaf8c56[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:53,039 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:53,039 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,040 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:53,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:53,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:53,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:53,048 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:53,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:53,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:53,051 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:53,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:53,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:53,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:53,052 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:53,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:53,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:53,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:53,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:53,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:53,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:53,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:53,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:53,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:53,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:53,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:53,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:53,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:53,170 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:53,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:53,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:53,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:53,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:53,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:53,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:53,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:53,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:53,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:53,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:53,175 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:53,175 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:53,175 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:53,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:53,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:53,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:53,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036893270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036893272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036893273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036893374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036893377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036893378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036893581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036893582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,585 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/e109f765a48648aeb137f1be11d96eb6 2023-03-28 20:53:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036893582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036893884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036893886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:53,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036893886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,007 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/87dcb5eb6d0c49ba9076bf2bc9e90b06 2023-03-28 20:53:54,064 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1cbc1d4c73ca416db976d7f57666d87c 2023-03-28 20:53:54,071 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/e109f765a48648aeb137f1be11d96eb6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6 2023-03-28 20:53:54,081 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6, entries=150, sequenceid=332, filesize=11.9 K 2023-03-28 20:53:54,083 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/87dcb5eb6d0c49ba9076bf2bc9e90b06 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06 2023-03-28 20:53:54,093 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06, entries=150, sequenceid=332, filesize=11.9 K 2023-03-28 20:53:54,095 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1cbc1d4c73ca416db976d7f57666d87c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c 2023-03-28 20:53:54,101 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c, entries=150, sequenceid=332, filesize=11.9 K 2023-03-28 20:53:54,103 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 59dd79967ef9e13a346f2f7f11b173ef in 928ms, sequenceid=332, compaction requested=true 2023-03-28 20:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:54,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:54,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:54,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674c3137[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:54,114 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:54,114 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,115 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:54,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:54,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:54,121 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:54,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:54,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:54,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:54,121 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:54,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:54,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:54,125 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,127 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:54,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:54,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:54,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:54,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:54,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:54,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:54,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:54,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,134 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:54,134 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:54,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:54,234 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:54,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:54,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:54,239 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:54,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:54,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:54,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:54,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:54,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:54,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:54,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:54,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:54,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:54,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:54,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:54,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:54,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:54,245 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:53:54,245 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:54,245 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:54,245 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:54,246 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:54,246 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:54,246 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:54,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:54,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:54,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036894426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036894428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036894432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036894536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036894536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036894537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,663 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/ae4c58738b9741418ae806b7e7375c85 2023-03-28 20:53:54,687 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/f79a99a2cb2f4505a5d3012f94cd5ff9 2023-03-28 20:53:54,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036894739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:54,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036894741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:54,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036894741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036895046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036895047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036895049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,110 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/b2a070d718e94087b3c5efab5e8d5ecc 2023-03-28 20:53:55,116 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/ae4c58738b9741418ae806b7e7375c85 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85 2023-03-28 20:53:55,122 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85, entries=150, sequenceid=347, filesize=11.9 K 2023-03-28 20:53:55,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/f79a99a2cb2f4505a5d3012f94cd5ff9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9 2023-03-28 20:53:55,129 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9, entries=150, sequenceid=347, filesize=11.9 K 2023-03-28 20:53:55,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/b2a070d718e94087b3c5efab5e8d5ecc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc 2023-03-28 20:53:55,135 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc, entries=150, sequenceid=347, filesize=11.9 K 2023-03-28 20:53:55,136 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 59dd79967ef9e13a346f2f7f11b173ef in 892ms, sequenceid=347, compaction requested=true 2023-03-28 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e45012a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,143 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,143 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,151 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,151 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,153 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,155 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,157 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,158 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,158 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:55,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:55,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:55,262 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:55,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:55,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:55,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:55,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:55,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:55,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:55,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:55,284 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:53:55,284 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,299 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/cb8300053139494eab5904a9a49bcc18 2023-03-28 20:53:55,313 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/38547ea03f0b4c39915f91e2ed18bf3a 2023-03-28 20:53:55,326 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ccbdce7177b14b0fafc65bbdebed5561 2023-03-28 20:53:55,332 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/cb8300053139494eab5904a9a49bcc18 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18 2023-03-28 20:53:55,339 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18, entries=150, sequenceid=364, filesize=11.9 K 2023-03-28 20:53:55,340 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/38547ea03f0b4c39915f91e2ed18bf3a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a 2023-03-28 20:53:55,346 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a, entries=150, sequenceid=364, filesize=11.9 K 2023-03-28 20:53:55,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/ccbdce7177b14b0fafc65bbdebed5561 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561 2023-03-28 20:53:55,352 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561, entries=150, sequenceid=364, filesize=11.9 K 2023-03-28 20:53:55,354 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 59dd79967ef9e13a346f2f7f11b173ef in 70ms, sequenceid=364, compaction requested=true 2023-03-28 20:53:55,354 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:55,354 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,358 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353b49a4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,367 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,370 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,371 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,380 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,380 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,381 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,381 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,381 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,481 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:55,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:55,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:55,485 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:55,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:55,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:55,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,488 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:55,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:55,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:55,493 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,493 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:55,493 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:55,493 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f4ae69[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,498 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,503 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,503 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,508 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,510 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:55,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,512 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,512 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:55,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:53:55,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:55,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:55,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:55,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:55,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/74ccb69c32a445968952cfd1574b27d8 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680036835587, max timestamp=1680036835601Num uniques 859; 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680036835587, max timestamp=1680036835601Num uniques 859; 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680036835587, max timestamp=1680036835601Num uniques 859; 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-28 20:53:55,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-28 20:53:55,611 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:55,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:55,616 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:55,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:55,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:55,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:55,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:55,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:55,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:55,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:55,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,622 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3137f429[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,627 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,627 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,628 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,632 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,632 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,634 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,634 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,634 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,635 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:55,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036895653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036895653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036895653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:55,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:55,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:55,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:55,753 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:55,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:55,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:55,754 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,754 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,755 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:55,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:55,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036895758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,760 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036895759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036895759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672dff74[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,763 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,763 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,766 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,769 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,770 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,775 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,775 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,775 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:55,875 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:55,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:55,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:55,879 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:55,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,881 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:55,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:55,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:55,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:55,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:55,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@834fc5c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:55,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:55,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,889 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:55,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:55,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:55,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:55,893 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:55,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:55,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:55,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:55,893 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:55,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:55,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:55,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,896 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:55,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:55,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:55,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:55,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:55,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:55,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:55,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:55,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,901 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:55,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:55,901 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:55,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:55,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036895961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036895962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:55,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036895963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:55,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/6a1fea0f7ac74a0ab9cefcfd0c46f977 2023-03-28 20:53:56,001 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,005 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,014 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/d3fbb0b433c2414488b2f3423bc34019 2023-03-28 20:53:56,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,017 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,017 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,017 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,017 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,019 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa18cd5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,024 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/74ccb69c32a445968952cfd1574b27d8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8 2023-03-28 20:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,033 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,034 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8, entries=200, sequenceid=376, filesize=14.3 K 2023-03-28 20:53:56,035 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/6a1fea0f7ac74a0ab9cefcfd0c46f977 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977 2023-03-28 20:53:56,035 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,038 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977, entries=150, sequenceid=376, filesize=11.9 K 2023-03-28 20:53:56,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/d3fbb0b433c2414488b2f3423bc34019 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,044 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019, entries=150, sequenceid=376, filesize=11.9 K 2023-03-28 20:53:56,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 59dd79967ef9e13a346f2f7f11b173ef in 483ms, sequenceid=376, compaction requested=true 2023-03-28 20:53:56,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:56,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:56,049 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:53:56,049 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:53:56,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:56,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:56,052 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 114882 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:53:56,052 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:56,052 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110002 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:53:56,052 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,052 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:56,052 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f567d71c53ab4f3a93f4c5c4a04a4182, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=112.2 K 2023-03-28 20:53:56,052 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,052 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/270cc39210d745269893cfa30851952c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=107.4 K 2023-03-28 20:53:56,053 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f567d71c53ab4f3a93f4c5c4a04a4182, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:56,053 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 270cc39210d745269893cfa30851952c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:56,053 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d3fd354eea054a0599c434139ba74c94, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680036829873 2023-03-28 20:53:56,054 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 871a06864d2f448d9cce1211b46c4606, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680036829873 2023-03-28 20:53:56,054 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c93b72e6515e44a7a6b400b4d06adb47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680036830505 2023-03-28 20:53:56,054 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting eb346804f8194fd4b59ecd1b8d9ea51a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680036830505 2023-03-28 20:53:56,054 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 78e71b69bf7a4de9ab2ad8bbf6130e63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680036830850 2023-03-28 20:53:56,055 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 14cdd40ac5434eb08d2d58ccc4b37b21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680036830850 2023-03-28 20:53:56,055 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e96caf793bf541639343145171d1ce7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036831995 2023-03-28 20:53:56,055 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting dc207d577f4043b687b46dc653b4b1ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036831995 2023-03-28 20:53:56,055 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 87dcb5eb6d0c49ba9076bf2bc9e90b06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680036832623 2023-03-28 20:53:56,056 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e109f765a48648aeb137f1be11d96eb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680036832623 2023-03-28 20:53:56,056 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f79a99a2cb2f4505a5d3012f94cd5ff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680036833269 2023-03-28 20:53:56,056 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ae4c58738b9741418ae806b7e7375c85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680036833269 2023-03-28 20:53:56,056 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 38547ea03f0b4c39915f91e2ed18bf3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680036834425 2023-03-28 20:53:56,056 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cb8300053139494eab5904a9a49bcc18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680036834425 2023-03-28 20:53:56,057 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6a1fea0f7ac74a0ab9cefcfd0c46f977, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835562 2023-03-28 20:53:56,057 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 74ccb69c32a445968952cfd1574b27d8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835561 2023-03-28 20:53:56,076 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#69 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:56,077 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#70 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:56,090 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/733d717032a04a688fc98240137b6323 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/733d717032a04a688fc98240137b6323 2023-03-28 20:53:56,092 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/5b79a694de5b414e94469c7852d7351a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/5b79a694de5b414e94469c7852d7351a 2023-03-28 20:53:56,097 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 733d717032a04a688fc98240137b6323(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:56,097 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:56,097 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=7, startTime=1680036836049; duration=0sec 2023-03-28 20:53:56,097 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:56,097 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:53:56,098 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 5b79a694de5b414e94469c7852d7351a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:56,098 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:56,098 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=7, startTime=1680036836049; duration=0sec 2023-03-28 20:53:56,098 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:56,100 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110002 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:53:56,100 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:56,100 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,100 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c51d222c0d2342fb95683d6fb85f2ac0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=107.4 K 2023-03-28 20:53:56,100 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c51d222c0d2342fb95683d6fb85f2ac0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680036829746 2023-03-28 20:53:56,101 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting dd948e74c24d4eb78658cc9aa743db21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680036829873 2023-03-28 20:53:56,101 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5642388daf3c44d09bf46076d140a753, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680036830505 2023-03-28 20:53:56,102 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 67ca5113b76a4caf81cb49b23232ebe8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680036830850 2023-03-28 20:53:56,102 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ff1ddbc858764ece983af3327e4d5613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036831995 2023-03-28 20:53:56,102 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1cbc1d4c73ca416db976d7f57666d87c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680036832623 2023-03-28 20:53:56,103 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b2a070d718e94087b3c5efab5e8d5ecc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680036833269 2023-03-28 20:53:56,103 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ccbdce7177b14b0fafc65bbdebed5561, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680036834425 2023-03-28 20:53:56,104 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d3fbb0b433c2414488b2f3423bc34019, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835562 2023-03-28 20:53:56,119 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#71 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:56,140 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/e0a4ea1c64134efc936dd2ba98113d0a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/e0a4ea1c64134efc936dd2ba98113d0a 2023-03-28 20:53:56,145 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,147 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into e0a4ea1c64134efc936dd2ba98113d0a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:56,147 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:56,147 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=7, startTime=1680036836049; duration=0sec 2023-03-28 20:53:56,147 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:56,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,149 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,152 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,154 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-03-28 20:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:56,155 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:56,162 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/17fdff9b24d14b52ad17a6dcbd68dc22 2023-03-28 20:53:56,176 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/446121b3b4004b8bb910d26049265a6b 2023-03-28 20:53:56,213 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1ddb77c9bb3249678f2a2c06d8f53984 2023-03-28 20:53:56,219 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/17fdff9b24d14b52ad17a6dcbd68dc22 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22 2023-03-28 20:53:56,226 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22, entries=150, sequenceid=411, filesize=11.9 K 2023-03-28 20:53:56,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/446121b3b4004b8bb910d26049265a6b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b 2023-03-28 20:53:56,233 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b, entries=150, sequenceid=411, filesize=11.9 K 2023-03-28 20:53:56,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/1ddb77c9bb3249678f2a2c06d8f53984 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984 2023-03-28 20:53:56,240 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984, entries=150, sequenceid=411, filesize=11.9 K 2023-03-28 20:53:56,242 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=0 B/0 for 59dd79967ef9e13a346f2f7f11b173ef in 88ms, sequenceid=411, compaction requested=false 2023-03-28 20:53:56,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:56,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,244 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1439b514[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,247 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,247 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,251 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,254 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,254 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,260 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,264 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,264 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:56,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:56,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/fd457bf870cb46e59982e770446c7f46 2023-03-28 20:53:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036896309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036836303, max timestamp=1680036836309Num uniques 859; 2023-03-28 20:53:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:53:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036836303, max timestamp=1680036836309Num uniques 859; 2023-03-28 20:53:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:53:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036836303, max timestamp=1680036836309Num uniques 859; 2023-03-28 20:53:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:53:56,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036896350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036896352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,364 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,369 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,377 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,379 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c672eb8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,381 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,381 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,385 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,385 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,385 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,385 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,391 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,391 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,391 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,391 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,392 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,394 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,394 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,395 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:56,402 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,403 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036896419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036896455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036896455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,503 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,506 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,507 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,507 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,508 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,510 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,510 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,510 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,510 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,512 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,513 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,513 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339ee774[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,517 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,520 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,524 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,524 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,526 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,526 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,526 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,529 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,531 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036896622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,636 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,642 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,643 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,643 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,644 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,646 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,646 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cf02f6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,655 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,655 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036896657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,658 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,658 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,661 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036896659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,665 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,665 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,665 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/8a661a3f2b8544c08cb92e1728a15a04 2023-03-28 20:53:56,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,770 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,772 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,782 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,782 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,784 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,784 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,784 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,784 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c07924e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,790 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,790 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,790 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,791 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,794 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,794 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,796 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,796 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,796 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,800 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,802 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,902 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:56,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:56,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:56,905 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:56,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:56,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:56,906 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,906 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,908 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:56,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:56,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59307198[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:56,915 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:56,915 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:56,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:56,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:56,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:56,926 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:56,926 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:56,926 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:56,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:56,927 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:56,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036896926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,930 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:56,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,935 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:56,935 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:56,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:56,936 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:56,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:56,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:56,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:56,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:56,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:56,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036896961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:56,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:56,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036896964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:57,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:57,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:57,060 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:57,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:57,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:57,061 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:57,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:57,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:57,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:57,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:57,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:57,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d51888d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:57,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:57,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:57,078 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,078 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:57,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:57,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:57,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:57,083 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:57,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:57,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:57,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:57,083 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:57,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:57,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:57,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,091 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:57,091 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:57,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/4a9dfcfa69444d458413324c73566a4f 2023-03-28 20:53:57,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/fd457bf870cb46e59982e770446c7f46 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46 2023-03-28 20:53:57,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46, entries=150, sequenceid=422, filesize=11.9 K 2023-03-28 20:53:57,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/8a661a3f2b8544c08cb92e1728a15a04 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04 2023-03-28 20:53:57,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04, entries=150, sequenceid=422, filesize=11.9 K 2023-03-28 20:53:57,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/4a9dfcfa69444d458413324c73566a4f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f 2023-03-28 20:53:57,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f, entries=150, sequenceid=422, filesize=11.9 K 2023-03-28 20:53:57,153 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 59dd79967ef9e13a346f2f7f11b173ef in 877ms, sequenceid=422, compaction requested=true 2023-03-28 20:53:57,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:57,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:57,153 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:57,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:57,153 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:57,154 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:57,154 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:57,154 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:57,154 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:57,155 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,155 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/733d717032a04a688fc98240137b6323, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.5 K 2023-03-28 20:53:57,155 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/5b79a694de5b414e94469c7852d7351a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.5 K 2023-03-28 20:53:57,155 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 733d717032a04a688fc98240137b6323, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835562 2023-03-28 20:53:57,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5b79a694de5b414e94469c7852d7351a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835562 2023-03-28 20:53:57,156 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 17fdff9b24d14b52ad17a6dcbd68dc22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680036835622 2023-03-28 20:53:57,156 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 446121b3b4004b8bb910d26049265a6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680036835622 2023-03-28 20:53:57,156 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fd457bf870cb46e59982e770446c7f46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:53:57,156 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8a661a3f2b8544c08cb92e1728a15a04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:53:57,163 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:57,164 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:57,184 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/c86749fd9afb4223962ee689a222d94e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c86749fd9afb4223962ee689a222d94e 2023-03-28 20:53:57,191 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:57,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:57,192 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into c86749fd9afb4223962ee689a222d94e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:57,192 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,192 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036837153; duration=0sec 2023-03-28 20:53:57,192 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:57,192 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:57,194 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:57,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:57,194 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:53:57,195 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,195 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/e0a4ea1c64134efc936dd2ba98113d0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.5 K 2023-03-28 20:53:57,195 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e0a4ea1c64134efc936dd2ba98113d0a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680036835562 2023-03-28 20:53:57,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:57,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:57,196 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1ddb77c9bb3249678f2a2c06d8f53984, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680036835622 2023-03-28 20:53:57,196 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:57,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:57,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:57,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,197 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4a9dfcfa69444d458413324c73566a4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:53:57,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:57,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:57,201 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,201 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:57,201 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,201 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:57,201 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:57,202 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,209 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:57,222 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/cd2204a82d4943b383e32d7be58a6900 2023-03-28 20:53:57,242 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/dfe508823a2447829f2c5e891107f486 2023-03-28 20:53:57,260 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/7277e88225a74cbf97a3f9d78dc69cd6 2023-03-28 20:53:57,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/cd2204a82d4943b383e32d7be58a6900 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900 2023-03-28 20:53:57,274 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900, entries=150, sequenceid=459, filesize=11.9 K 2023-03-28 20:53:57,275 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/dfe508823a2447829f2c5e891107f486 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486 2023-03-28 20:53:57,282 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486, entries=150, sequenceid=459, filesize=11.9 K 2023-03-28 20:53:57,282 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/7277e88225a74cbf97a3f9d78dc69cd6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6 2023-03-28 20:53:57,288 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6, entries=150, sequenceid=459, filesize=11.9 K 2023-03-28 20:53:57,288 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 59dd79967ef9e13a346f2f7f11b173ef in 87ms, sequenceid=459, compaction requested=false 2023-03-28 20:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:57,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4054e163[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:57,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:57,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:57,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:57,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:57,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:57,306 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:57,306 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:57,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:57,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,314 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,314 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,414 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:57,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:57,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:57,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:57,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:57,418 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:57,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:57,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:57,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:57,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:57,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:57,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:57,422 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,423 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:57,423 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,423 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:57,425 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:57,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ec5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:57,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:57,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:57,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:57,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:57,434 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:57,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:57,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:57,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:57,434 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:57,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:57,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:57,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:57,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:57,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:57,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:57,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:57,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:57,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/61fa9f3b9cef41e8841a4a9161a70522 2023-03-28 20:53:57,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036837484, max timestamp=1680036837505Num uniques 859; 2023-03-28 20:53:57,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:53:57,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036837484, max timestamp=1680036837505Num uniques 859; 2023-03-28 20:53:57,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036837484, max timestamp=1680036837505Num uniques 859; 2023-03-28 20:53:57,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:53:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:53:57,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cbd96bf2ac3b444dbe6588871e34d610 2023-03-28 20:53:57,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/c325c1eed7bc4882952c7cdc1917f75a 2023-03-28 20:53:57,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/61fa9f3b9cef41e8841a4a9161a70522 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522 2023-03-28 20:53:57,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522, entries=150, sequenceid=470, filesize=11.9 K 2023-03-28 20:53:57,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cbd96bf2ac3b444dbe6588871e34d610 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610 2023-03-28 20:53:57,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610, entries=100, sequenceid=470, filesize=9.5 K 2023-03-28 20:53:57,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/c325c1eed7bc4882952c7cdc1917f75a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a 2023-03-28 20:53:57,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036897537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036897539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:57,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036897544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:57,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:57,556 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:57,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:57,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:57,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:57,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:57,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:57,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:57,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a, entries=100, sequenceid=470, filesize=9.5 K 2023-03-28 20:53:57,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 59dd79967ef9e13a346f2f7f11b173ef in 107ms, sequenceid=470, compaction requested=true 2023-03-28 20:53:57,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-28 20:53:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-28 20:53:57,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:57,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,568 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:57,569 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:57,599 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/2459ea5c9ff04b39b1637e3eebe69ec4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2459ea5c9ff04b39b1637e3eebe69ec4 2023-03-28 20:53:57,611 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 2459ea5c9ff04b39b1637e3eebe69ec4(size=12.8 K), total size for store is 34.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:57,611 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,611 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036837153; duration=0sec 2023-03-28 20:53:57,611 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-28 20:53:57,611 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-28 20:53:57,613 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:53:57,613 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:53:57,613 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. because compaction request was cancelled 2023-03-28 20:53:57,613 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:57,614 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:57,615 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:53:57,615 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,615 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2459ea5c9ff04b39b1637e3eebe69ec4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=34.2 K 2023-03-28 20:53:57,615 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2459ea5c9ff04b39b1637e3eebe69ec4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:53:57,616 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting dfe508823a2447829f2c5e891107f486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680036836341 2023-03-28 20:53:57,616 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cbd96bf2ac3b444dbe6588871e34d610, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680036837447 2023-03-28 20:53:57,625 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/02a797bf329e42629dd1b224e999aae8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/02a797bf329e42629dd1b224e999aae8 2023-03-28 20:53:57,632 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into 02a797bf329e42629dd1b224e999aae8(size=12.8 K), total size for store is 34.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:57,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:57,632 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=13, startTime=1680036837153; duration=0sec 2023-03-28 20:53:57,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:53:57,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:53:57,633 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37440 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:53:57,633 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:53:57,634 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:57,634 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c86749fd9afb4223962ee689a222d94e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.6 K 2023-03-28 20:53:57,634 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c86749fd9afb4223962ee689a222d94e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:53:57,635 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cd2204a82d4943b383e32d7be58a6900, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680036836341 2023-03-28 20:53:57,635 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 61fa9f3b9cef41e8841a4a9161a70522, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680036837447 2023-03-28 20:53:57,637 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:57,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:57,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:57,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036897664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036897668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,673 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:53:57,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036897677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036897770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036897770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036897782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036897975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036897975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:57,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:57,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036897985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,004 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/91ed9be0cac2479ba7c43cde8d985865 2023-03-28 20:53:58,023 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/c878b60a2bcc492799ad4c9ff5cb463c 2023-03-28 20:53:58,091 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/ab6c13c423504c9caeb76ffc5f90be5f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/ab6c13c423504c9caeb76ffc5f90be5f 2023-03-28 20:53:58,098 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into ab6c13c423504c9caeb76ffc5f90be5f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:58,098 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:58,098 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036837562; duration=0sec 2023-03-28 20:53:58,098 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:58,101 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/eb74de937220477b983332a45ba46240 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb74de937220477b983332a45ba46240 2023-03-28 20:53:58,106 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into eb74de937220477b983332a45ba46240(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:53:58,107 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:58,107 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036837562; duration=0sec 2023-03-28 20:53:58,107 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:53:58,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036898279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036898281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036898290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,435 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/aa1b6888830247e89318796d45da663c 2023-03-28 20:53:58,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/91ed9be0cac2479ba7c43cde8d985865 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865 2023-03-28 20:53:58,447 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865, entries=150, sequenceid=505, filesize=11.9 K 2023-03-28 20:53:58,448 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/c878b60a2bcc492799ad4c9ff5cb463c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c 2023-03-28 20:53:58,452 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c, entries=150, sequenceid=505, filesize=11.9 K 2023-03-28 20:53:58,453 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/aa1b6888830247e89318796d45da663c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c 2023-03-28 20:53:58,458 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c, entries=150, sequenceid=505, filesize=11.9 K 2023-03-28 20:53:58,459 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for 59dd79967ef9e13a346f2f7f11b173ef in 891ms, sequenceid=505, compaction requested=true 2023-03-28 20:53:58,459 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:58,459 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:58,461 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:58,462 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,462 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae783ec[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:58,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:58,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,467 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:58,467 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,467 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:58,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:58,475 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:58,475 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:58,475 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:58,475 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:58,475 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:58,476 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,476 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:58,476 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:58,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,478 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:58,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:58,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:58,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:58,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:58,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:58,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:58,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:58,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:58,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:58,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:58,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:58,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:58,589 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:58,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:58,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:58,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:58,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:58,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:58,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:58,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:58,592 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:53:58,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:58,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:58,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:58,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:58,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:58,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:53:58,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:58,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:58,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036898817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036838806, max timestamp=1680036838818Num uniques 859; 2023-03-28 20:53:58,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:53:58,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036838806, max timestamp=1680036838818Num uniques 859; 2023-03-28 20:53:58,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036838806, max timestamp=1680036838818Num uniques 859; 2023-03-28 20:53:58,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:53:58,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:53:58,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036898847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036898849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036898921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036898950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:58,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:58,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036898954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,013 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/f0d0f2c882454a54ae66baa85e9d4d14 2023-03-28 20:53:59,048 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cb6584f9ba2a42c192c08f521be385aa 2023-03-28 20:53:59,063 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/a948b7dc76424cc8a5c5d641adc1563a 2023-03-28 20:53:59,071 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/f0d0f2c882454a54ae66baa85e9d4d14 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14 2023-03-28 20:53:59,076 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14, entries=100, sequenceid=520, filesize=9.5 K 2023-03-28 20:53:59,077 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cb6584f9ba2a42c192c08f521be385aa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa 2023-03-28 20:53:59,088 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa, entries=100, sequenceid=520, filesize=9.5 K 2023-03-28 20:53:59,089 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/a948b7dc76424cc8a5c5d641adc1563a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a 2023-03-28 20:53:59,105 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a, entries=100, sequenceid=520, filesize=9.5 K 2023-03-28 20:53:59,106 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 59dd79967ef9e13a346f2f7f11b173ef in 514ms, sequenceid=520, compaction requested=true 2023-03-28 20:53:59,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:53:59,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,109 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,109 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b637ddd[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,113 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,113 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,116 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,119 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,119 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:53:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:53:59,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:53:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:53:59,128 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/342f16c47cf04ac7a655f4d4d8c3a498 2023-03-28 20:53:59,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036899155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036899158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036899159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,232 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,243 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,245 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,245 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,246 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,246 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,247 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,247 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,248 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,248 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5bc22b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,254 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,265 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,265 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036899262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,270 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,274 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:59,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,384 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,389 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,389 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,389 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,389 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202006c3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,394 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,394 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,395 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,402 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,402 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,402 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,405 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,405 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,407 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,407 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,408 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,408 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:59,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036899465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036899465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036899468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,520 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,521 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,521 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,523 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,535 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46be09c8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,540 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,544 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,544 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,549 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,554 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,554 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:59,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/584c91a9b2be4347bf2f53332a9317dc 2023-03-28 20:53:59,655 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,659 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,662 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,663 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,663 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e30957[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,696 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,696 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,710 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036899777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,814 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,824 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dbdb3a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,831 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,834 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,834 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,836 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,836 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:59,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,853 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,853 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:53:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:53:59,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:53:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:53:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:53:59,961 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:53:59,961 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,961 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:53:59,961 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:53:59,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:53:59,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:53:59,965 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,965 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:53:59,965 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:53:59,965 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:53:59,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:53:59,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:53:59,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446ddf7c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:53:59,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,970 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:53:59,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:53:59,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:53:59,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:53:59,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036899970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:53:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036899973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:53:59,975 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:53:59,976 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:53:59,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:53:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,979 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:53:59,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:53:59,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:53:59,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:53:59,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:53:59,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:53:59,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:53:59,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:53:59,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:53:59,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:53:59,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:53:59,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:53:59,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:53:59,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/60a2a7d6a08e4c70a8213d6527a90f11 2023-03-28 20:54:00,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/342f16c47cf04ac7a655f4d4d8c3a498 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498 2023-03-28 20:54:00,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498, entries=150, sequenceid=556, filesize=11.9 K 2023-03-28 20:54:00,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/584c91a9b2be4347bf2f53332a9317dc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc 2023-03-28 20:54:00,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc, entries=150, sequenceid=556, filesize=11.9 K 2023-03-28 20:54:00,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/60a2a7d6a08e4c70a8213d6527a90f11 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11 2023-03-28 20:54:00,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11, entries=150, sequenceid=556, filesize=11.9 K 2023-03-28 20:54:00,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 59dd79967ef9e13a346f2f7f11b173ef in 956ms, sequenceid=556, compaction requested=true 2023-03-28 20:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:00,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:00,082 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:00,082 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:00,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:00,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:00,082 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:00,083 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:00,083 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:54:00,083 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:00,083 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,083 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:54:00,083 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb74de937220477b983332a45ba46240, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=46.2 K 2023-03-28 20:54:00,083 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,084 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/ab6c13c423504c9caeb76ffc5f90be5f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=46.2 K 2023-03-28 20:54:00,084 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting eb74de937220477b983332a45ba46240, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680036836344 2023-03-28 20:54:00,084 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ab6c13c423504c9caeb76ffc5f90be5f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680036836344 2023-03-28 20:54:00,085 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 91ed9be0cac2479ba7c43cde8d985865, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680036837526 2023-03-28 20:54:00,085 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c878b60a2bcc492799ad4c9ff5cb463c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680036837526 2023-03-28 20:54:00,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:00,085 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f0d0f2c882454a54ae66baa85e9d4d14, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680036837666 2023-03-28 20:54:00,085 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cb6584f9ba2a42c192c08f521be385aa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680036837666 2023-03-28 20:54:00,085 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 342f16c47cf04ac7a655f4d4d8c3a498, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:00,086 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 584c91a9b2be4347bf2f53332a9317dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:00,098 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#98 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:00,098 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:00,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:00,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:00,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,099 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:00,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:00,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:00,106 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:00,107 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:54:00,107 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:00,107 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:54:00,107 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:00,135 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/958cfde2834a4f05b48b8d878879a3db as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/958cfde2834a4f05b48b8d878879a3db 2023-03-28 20:54:00,142 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 958cfde2834a4f05b48b8d878879a3db(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:00,143 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:00,143 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=12, startTime=1680036840081; duration=0sec 2023-03-28 20:54:00,143 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:00,143 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:54:00,146 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69114 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:54:00,146 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:54:00,146 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,146 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/02a797bf329e42629dd1b224e999aae8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=67.5 K 2023-03-28 20:54:00,147 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 02a797bf329e42629dd1b224e999aae8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680036836271 2023-03-28 20:54:00,147 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7277e88225a74cbf97a3f9d78dc69cd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680036836341 2023-03-28 20:54:00,148 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c325c1eed7bc4882952c7cdc1917f75a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680036837447 2023-03-28 20:54:00,149 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting aa1b6888830247e89318796d45da663c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680036837526 2023-03-28 20:54:00,149 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a948b7dc76424cc8a5c5d641adc1563a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680036837666 2023-03-28 20:54:00,150 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 60a2a7d6a08e4c70a8213d6527a90f11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:00,158 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/8644b97af5d34425a76cd4f19562459c 2023-03-28 20:54:00,163 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#101 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:00,192 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cde195ba14e2418bacd0c3e76456cf74 2023-03-28 20:54:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:00,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036840332, max timestamp=1680036840354Num uniques 859; 2023-03-28 20:54:00,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036840332, max timestamp=1680036840354Num uniques 859; 2023-03-28 20:54:00,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:00,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:00,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036840332, max timestamp=1680036840354Num uniques 859; 2023-03-28 20:54:00,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:00,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036900414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,518 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/8603a441fcab47ca8dcaf1280fb7b8f8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8603a441fcab47ca8dcaf1280fb7b8f8 2023-03-28 20:54:00,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036900517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,525 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 8603a441fcab47ca8dcaf1280fb7b8f8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:00,525 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:00,525 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=12, startTime=1680036840082; duration=0sec 2023-03-28 20:54:00,525 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:00,592 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/d8dd5cc9ae6449288ffdaa5fa9e408af as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d8dd5cc9ae6449288ffdaa5fa9e408af 2023-03-28 20:54:00,604 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into d8dd5cc9ae6449288ffdaa5fa9e408af(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:00,605 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:00,605 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=10, startTime=1680036840082; duration=0sec 2023-03-28 20:54:00,605 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:00,606 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/63c8810b56b844b0b47f80d5dd8cb2de 2023-03-28 20:54:00,612 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/8644b97af5d34425a76cd4f19562459c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c 2023-03-28 20:54:00,617 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c, entries=150, sequenceid=566, filesize=11.9 K 2023-03-28 20:54:00,618 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/cde195ba14e2418bacd0c3e76456cf74 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74 2023-03-28 20:54:00,622 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74, entries=150, sequenceid=566, filesize=11.9 K 2023-03-28 20:54:00,623 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/63c8810b56b844b0b47f80d5dd8cb2de as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de 2023-03-28 20:54:00,627 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de, entries=150, sequenceid=566, filesize=11.9 K 2023-03-28 20:54:00,628 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 59dd79967ef9e13a346f2f7f11b173ef in 522ms, sequenceid=566, compaction requested=false 2023-03-28 20:54:00,628 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:00,628 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:00,629 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:00,629 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:00,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:00,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ab7e70[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:00,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:00,635 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:00,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:00,639 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:00,641 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:00,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:00,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:00,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:00,641 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:00,643 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,643 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,643 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,644 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,645 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:00,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:00,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,648 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:00,648 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,647 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:00,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-28 20:54:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:54:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:54:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:54:00,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:00,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036900743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,748 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:00,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:00,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:00,756 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:00,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:00,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:00,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:00,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:00,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:00,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:00,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:00,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:00,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:00,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0c78c3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:00,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:00,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,770 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:00,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:00,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:00,772 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:00,776 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:00,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:00,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:00,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:00,776 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:00,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,782 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:00,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:00,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:00,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,789 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:00,789 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,789 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,789 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,789 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036900846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,889 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:00,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:00,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:00,893 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:00,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:00,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:00,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:00,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:00,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:00,906 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,906 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:00,906 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:00,907 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:00,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b46e73[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:00,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:00,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,913 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:00,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:00,919 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:00,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:00,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:00,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:00,919 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:00,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:00,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,928 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:00,928 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:00,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:00,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:00,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:00,928 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:00,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:00,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:00,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38742 deadline: 1680036900974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:00,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:00,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38752 deadline: 1680036900977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:01,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38786 deadline: 1680036901014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,019 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18297 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:01,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,034 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,035 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:01,047 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c88cfe6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:01,049 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:01,049 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,051 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:01,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:01,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:01,053 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036901049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:01,058 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:01,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:01,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:01,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:01,058 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:01,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:01,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:01,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:01,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:01,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38772 deadline: 1680036901077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,081 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18342 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:01,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/ad0555f216804f2e80a89d7774db9718 2023-03-28 20:54:01,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,171 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,173 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/7b855c0129e742fc82f331d717c3da32 2023-03-28 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:01,178 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,178 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,178 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,178 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457974f9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:01,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:01,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:01,183 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:01,190 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:01,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:01,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:01,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:01,190 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:01,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:01,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:01,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:01,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,303 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,305 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,305 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,305 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,305 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,307 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,307 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,307 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,307 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:01,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,309 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2cd6f5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:01,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:01,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,316 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:01,317 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,317 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:01,321 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:01,321 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:01,321 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:01,321 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:01,321 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:01,323 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,323 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,323 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,325 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:01,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,328 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,329 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:01,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:01,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:01,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036901354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,432 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:01,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:01,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f0efc35[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:01,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:01,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:01,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:01,446 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:01,446 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:01,446 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:01,446 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:01,446 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:01,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,448 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,448 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,449 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,451 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:01,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,454 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:01,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:01,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,455 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,559 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,560 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,563 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:01,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:01,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:01,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:01,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:01,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1553f3cd[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:01,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:01,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:01,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:01,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:01,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:01,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:01,570 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,570 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:01,572 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:01,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:01,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:01,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:01,573 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:01,575 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,575 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:01,575 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:01,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,580 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:01,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:01,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,581 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:01,581 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:01,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:01,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/f8f3317cc4534e11960b180cb7a3edb2 2023-03-28 20:54:01,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/ad0555f216804f2e80a89d7774db9718 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718 2023-03-28 20:54:01,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718, entries=150, sequenceid=606, filesize=11.9 K 2023-03-28 20:54:01,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/7b855c0129e742fc82f331d717c3da32 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32 2023-03-28 20:54:01,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32, entries=150, sequenceid=606, filesize=11.9 K 2023-03-28 20:54:01,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/f8f3317cc4534e11960b180cb7a3edb2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2 2023-03-28 20:54:01,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2, entries=150, sequenceid=606, filesize=11.9 K 2023-03-28 20:54:01,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 59dd79967ef9e13a346f2f7f11b173ef in 890ms, sequenceid=606, compaction requested=true 2023-03-28 20:54:01,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:01,617 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:01,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:01,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:01,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:01,617 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:01,618 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:01,619 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:54:01,619 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,619 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/958cfde2834a4f05b48b8d878879a3db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.8 K 2023-03-28 20:54:01,619 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:01,619 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 958cfde2834a4f05b48b8d878879a3db, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:01,620 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:54:01,620 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,620 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8603a441fcab47ca8dcaf1280fb7b8f8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.8 K 2023-03-28 20:54:01,620 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8644b97af5d34425a76cd4f19562459c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680036839146 2023-03-28 20:54:01,620 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8603a441fcab47ca8dcaf1280fb7b8f8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:01,621 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ad0555f216804f2e80a89d7774db9718, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:01,621 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cde195ba14e2418bacd0c3e76456cf74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680036839146 2023-03-28 20:54:01,622 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7b855c0129e742fc82f331d717c3da32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:01,633 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#107 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:01,641 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:01,660 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/0b0e902ae7914c4ba8748b282ff6e951 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0b0e902ae7914c4ba8748b282ff6e951 2023-03-28 20:54:01,666 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 0b0e902ae7914c4ba8748b282ff6e951(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:01,666 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:01,666 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036841617; duration=0sec 2023-03-28 20:54:01,666 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:01,666 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:01,668 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:01,668 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:54:01,668 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,668 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d8dd5cc9ae6449288ffdaa5fa9e408af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.8 K 2023-03-28 20:54:01,668 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d8dd5cc9ae6449288ffdaa5fa9e408af, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680036838831 2023-03-28 20:54:01,669 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 63c8810b56b844b0b47f80d5dd8cb2de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680036839146 2023-03-28 20:54:01,670 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f8f3317cc4534e11960b180cb7a3edb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:01,680 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:01,681 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/eb4e0d650e0d44ba941c64ebc88d17a5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/eb4e0d650e0d44ba941c64ebc88d17a5 2023-03-28 20:54:01,681 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:01,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:01,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:01,688 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into eb4e0d650e0d44ba941c64ebc88d17a5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:01,688 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:01,688 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036841617; duration=0sec 2023-03-28 20:54:01,688 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:01,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:01,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:01,689 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:01,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:01,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:01,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:01,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:01,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:01,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:01,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:01,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:01,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:01,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:01,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:01,696 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:01,696 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. started... 2023-03-28 20:54:01,698 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:54:01,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:01,699 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:54:01,699 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:01,724 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/720437cde0f74bde89e591567ca68469 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/720437cde0f74bde89e591567ca68469 2023-03-28 20:54:01,732 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into 720437cde0f74bde89e591567ca68469(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:01,732 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:01,732 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=13, startTime=1680036841617; duration=0sec 2023-03-28 20:54:01,732 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:01,734 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/81925fde7e8b496f81fab6b22f4111e7 2023-03-28 20:54:01,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:01,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. as already flushing 2023-03-28 20:54:01,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036841880, max timestamp=1680036841915Num uniques 859; 2023-03-28 20:54:01,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:01,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036841880, max timestamp=1680036841915Num uniques 859; 2023-03-28 20:54:01,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:01,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036841880, max timestamp=1680036841915Num uniques 859; 2023-03-28 20:54:01,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:01,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:01,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036901967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,038 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x726fb449 to 127.0.0.1:56827 2023-03-28 20:54:02,038 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:02,039 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ae6c42b to 127.0.0.1:56827 2023-03-28 20:54:02,039 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:02,039 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75f5a5e6 to 127.0.0.1:56827 2023-03-28 20:54:02,039 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:02,040 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71f9f426 to 127.0.0.1:56827 2023-03-28 20:54:02,040 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:02,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:02,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036902072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,156 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/33fd1deac4ca4d0e9918053245ec3242 2023-03-28 20:54:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036902274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,566 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/b7499d878d5845cca0e8e4c2b662e93b 2023-03-28 20:54:02,571 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/81925fde7e8b496f81fab6b22f4111e7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7 2023-03-28 20:54:02,575 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7, entries=150, sequenceid=617, filesize=11.9 K 2023-03-28 20:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/33fd1deac4ca4d0e9918053245ec3242 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242 2023-03-28 20:54:02,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:02,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38756 deadline: 1680036902577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,580 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242, entries=150, sequenceid=617, filesize=11.9 K 2023-03-28 20:54:02,580 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/b7499d878d5845cca0e8e4c2b662e93b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b 2023-03-28 20:54:02,584 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b, entries=150, sequenceid=617, filesize=11.9 K 2023-03-28 20:54:02,585 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 59dd79967ef9e13a346f2f7f11b173ef in 887ms, sequenceid=617, compaction requested=false 2023-03-28 20:54:02,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:02,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:02,587 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b038322[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:02,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:02,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:02,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:02,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:02,593 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,593 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:02,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:02,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:02,597 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:02,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:02,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:02,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:02,597 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:02,599 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,599 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:02,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,599 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:02,599 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:02,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:02,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:02,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:02,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:02,604 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:02,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:02,604 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:02,604 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:02,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:02,704 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:02,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:02,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:02,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-03-28 20:54:02,987 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b41e306 to 127.0.0.1:56827 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:54:02,987 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:54:02,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:54:02,990 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377933e4 to 127.0.0.1:56827 2023-03-28 20:54:02,990 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:03,081 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e8196e0 to 127.0.0.1:56827 2023-03-28 20:54:03,081 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:03,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/c79e817ba8f248168a244c77d898a156 2023-03-28 20:54:03,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/7f67247a6385414b9a9f4e6953b3fdb3 2023-03-28 20:54:04,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/563c2ddb793d485985572e3ee1bf8169 2023-03-28 20:54:04,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/c79e817ba8f248168a244c77d898a156 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156 2023-03-28 20:54:04,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156, entries=150, sequenceid=656, filesize=11.9 K 2023-03-28 20:54:04,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/7f67247a6385414b9a9f4e6953b3fdb3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3 2023-03-28 20:54:04,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3, entries=150, sequenceid=656, filesize=11.9 K 2023-03-28 20:54:04,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/563c2ddb793d485985572e3ee1bf8169 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169 2023-03-28 20:54:04,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169, entries=150, sequenceid=656, filesize=11.9 K 2023-03-28 20:54:04,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=13.42 KB/13740 for 59dd79967ef9e13a346f2f7f11b173ef in 1281ms, sequenceid=656, compaction requested=true 2023-03-28 20:54:04,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:04,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:04,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:04,267 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:04,267 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:04,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:04,268 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:04,268 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:04,268 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/A is initiating minor compaction (all files) 2023-03-28 20:54:04,269 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/B is initiating minor compaction (all files) 2023-03-28 20:54:04,269 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/A in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:04,269 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/B in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:04,269 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0b0e902ae7914c4ba8748b282ff6e951, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.9 K 2023-03-28 20:54:04,269 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/eb4e0d650e0d44ba941c64ebc88d17a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.9 K 2023-03-28 20:54:04,269 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0b0e902ae7914c4ba8748b282ff6e951, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:04,269 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting eb4e0d650e0d44ba941c64ebc88d17a5, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:04,269 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 81925fde7e8b496f81fab6b22f4111e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680036840732 2023-03-28 20:54:04,269 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 33fd1deac4ca4d0e9918053245ec3242, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680036840732 2023-03-28 20:54:04,270 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c79e817ba8f248168a244c77d898a156, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1680036841962 2023-03-28 20:54:04,270 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7f67247a6385414b9a9f4e6953b3fdb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1680036841962 2023-03-28 20:54:04,277 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#A#compaction#116 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:04,279 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#B#compaction#117 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:04,687 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/04f9fe2a96ce415d8d883f07a74747f6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/04f9fe2a96ce415d8d883f07a74747f6 2023-03-28 20:54:04,688 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/63d42d21ee2a4479892fa6e06074b3ab as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/63d42d21ee2a4479892fa6e06074b3ab 2023-03-28 20:54:04,691 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/A of 59dd79967ef9e13a346f2f7f11b173ef into 04f9fe2a96ce415d8d883f07a74747f6(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:04,692 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:04,692 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/A, priority=13, startTime=1680036844267; duration=0sec 2023-03-28 20:54:04,692 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:04,692 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:04,692 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/B of 59dd79967ef9e13a346f2f7f11b173ef into 63d42d21ee2a4479892fa6e06074b3ab(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:04,692 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:04,692 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/B, priority=13, startTime=1680036844267; duration=0sec 2023-03-28 20:54:04,692 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:04,693 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:04,693 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 59dd79967ef9e13a346f2f7f11b173ef/C is initiating minor compaction (all files) 2023-03-28 20:54:04,693 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 59dd79967ef9e13a346f2f7f11b173ef/C in TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:04,693 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/720437cde0f74bde89e591567ca68469, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp, totalSize=36.9 K 2023-03-28 20:54:04,694 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 720437cde0f74bde89e591567ca68469, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1680036840411 2023-03-28 20:54:04,694 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b7499d878d5845cca0e8e4c2b662e93b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680036840732 2023-03-28 20:54:04,694 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 563c2ddb793d485985572e3ee1bf8169, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1680036841962 2023-03-28 20:54:04,700 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 59dd79967ef9e13a346f2f7f11b173ef#C#compaction#118 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:04,708 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/3750f2ebef5e4bd0b4952e349eb5247b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/3750f2ebef5e4bd0b4952e349eb5247b 2023-03-28 20:54:04,712 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 59dd79967ef9e13a346f2f7f11b173ef/C of 59dd79967ef9e13a346f2f7f11b173ef into 3750f2ebef5e4bd0b4952e349eb5247b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:04,712 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:04,712 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef., storeName=59dd79967ef9e13a346f2f7f11b173ef/C, priority=13, startTime=1680036844267; duration=0sec 2023-03-28 20:54:04,712 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:11,047 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2411a862 to 127.0.0.1:56827 2023-03-28 20:54:11,047 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:11,151 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ca4a062 to 127.0.0.1:56827 2023-03-28 20:54:11,151 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:11,151 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:54:11,151 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 152 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 139 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 207 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 3919 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 3991 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1760 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5269 rows 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1732 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5191 rows 2023-03-28 20:54:11,152 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:54:11,152 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d9637e to 127.0.0.1:56827 2023-03-28 20:54:11,152 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:11,158 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:54:11,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:54:11,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:11,171 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036851171"}]},"ts":"1680036851171"} 2023-03-28 20:54:11,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:11,172 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:54:11,258 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:54:11,260 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, UNASSIGN}] 2023-03-28 20:54:11,261 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, UNASSIGN 2023-03-28 20:54:11,262 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=59dd79967ef9e13a346f2f7f11b173ef, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:11,262 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036851262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036851262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036851262"}]},"ts":"1680036851262"} 2023-03-28 20:54:11,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:11,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 59dd79967ef9e13a346f2f7f11b173ef, disabling compactions & flushes 2023-03-28 20:54:11,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. after waiting 0 ms 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:11,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 59dd79967ef9e13a346f2f7f11b173ef 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=A 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=B 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 59dd79967ef9e13a346f2f7f11b173ef, store=C 2023-03-28 20:54:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:11,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:11,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:11,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/f31290739ceb47a89cdca7477e06e55a 2023-03-28 20:54:12,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/fbb56bf9f0e045a48d049f5038147ad6 2023-03-28 20:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:12,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/c4aefb719308410dbe318829184f6264 2023-03-28 20:54:12,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/A/f31290739ceb47a89cdca7477e06e55a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f31290739ceb47a89cdca7477e06e55a 2023-03-28 20:54:12,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f31290739ceb47a89cdca7477e06e55a, entries=150, sequenceid=666, filesize=11.9 K 2023-03-28 20:54:12,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/B/fbb56bf9f0e045a48d049f5038147ad6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fbb56bf9f0e045a48d049f5038147ad6 2023-03-28 20:54:12,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fbb56bf9f0e045a48d049f5038147ad6, entries=150, sequenceid=666, filesize=11.9 K 2023-03-28 20:54:12,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/.tmp/C/c4aefb719308410dbe318829184f6264 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c4aefb719308410dbe318829184f6264 2023-03-28 20:54:12,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c4aefb719308410dbe318829184f6264, entries=150, sequenceid=666, filesize=11.9 K 2023-03-28 20:54:12,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 59dd79967ef9e13a346f2f7f11b173ef in 1250ms, sequenceid=666, compaction requested=false 2023-03-28 20:54:12,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/65514e2b7d9340768c9ab0d7eca00608, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7af652d283654d1b9aa3a3196ba75c8c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fa49a995454d49a386c0ccb13675613a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f567d71c53ab4f3a93f4c5c4a04a4182, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/733d717032a04a688fc98240137b6323, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c86749fd9afb4223962ee689a222d94e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb74de937220477b983332a45ba46240, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/958cfde2834a4f05b48b8d878879a3db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0b0e902ae7914c4ba8748b282ff6e951, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156] to archive 2023-03-28 20:54:12,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:12,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/230c2c7b60f94544968671d2f8519a65 2023-03-28 20:54:12,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/be191d5da85340f9bd5529c10a9f1313 2023-03-28 20:54:12,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/10d1925c786b472b95c91b8d264f2986 2023-03-28 20:54:12,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/65514e2b7d9340768c9ab0d7eca00608 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/65514e2b7d9340768c9ab0d7eca00608 2023-03-28 20:54:12,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7f81a7b6e7f648999f375b4c00ef1ffa 2023-03-28 20:54:12,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d243d43439964ad29861840bdfd73bac 2023-03-28 20:54:12,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/138331c7117147efac3a7dd1d05e469d 2023-03-28 20:54:12,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7af652d283654d1b9aa3a3196ba75c8c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/7af652d283654d1b9aa3a3196ba75c8c 2023-03-28 20:54:12,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0cb28a788bf0441cba3e72aa830f749a 2023-03-28 20:54:12,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0942472af10146959169141f2c2df945 2023-03-28 20:54:12,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/bf147ba6c72d491889cc86fc1aa4539f 2023-03-28 20:54:12,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fa49a995454d49a386c0ccb13675613a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fa49a995454d49a386c0ccb13675613a 2023-03-28 20:54:12,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/5820c486cf8b41ceaa6f572eacd04756 2023-03-28 20:54:12,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f567d71c53ab4f3a93f4c5c4a04a4182 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f567d71c53ab4f3a93f4c5c4a04a4182 2023-03-28 20:54:12,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/1d7db8cddd4247cd89c77a0f3c03ead0 2023-03-28 20:54:12,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/d3fd354eea054a0599c434139ba74c94 2023-03-28 20:54:12,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb346804f8194fd4b59ecd1b8d9ea51a 2023-03-28 20:54:12,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/14cdd40ac5434eb08d2d58ccc4b37b21 2023-03-28 20:54:12,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/dc207d577f4043b687b46dc653b4b1ff 2023-03-28 20:54:12,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/e109f765a48648aeb137f1be11d96eb6 2023-03-28 20:54:12,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ae4c58738b9741418ae806b7e7375c85 2023-03-28 20:54:12,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cb8300053139494eab5904a9a49bcc18 2023-03-28 20:54:12,744 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/74ccb69c32a445968952cfd1574b27d8 2023-03-28 20:54:12,746 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/733d717032a04a688fc98240137b6323 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/733d717032a04a688fc98240137b6323 2023-03-28 20:54:12,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/17fdff9b24d14b52ad17a6dcbd68dc22 2023-03-28 20:54:12,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c86749fd9afb4223962ee689a222d94e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c86749fd9afb4223962ee689a222d94e 2023-03-28 20:54:12,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/fd457bf870cb46e59982e770446c7f46 2023-03-28 20:54:12,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/cd2204a82d4943b383e32d7be58a6900 2023-03-28 20:54:12,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb74de937220477b983332a45ba46240 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/eb74de937220477b983332a45ba46240 2023-03-28 20:54:12,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/61fa9f3b9cef41e8841a4a9161a70522 2023-03-28 20:54:12,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/91ed9be0cac2479ba7c43cde8d985865 2023-03-28 20:54:12,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f0d0f2c882454a54ae66baa85e9d4d14 2023-03-28 20:54:12,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/958cfde2834a4f05b48b8d878879a3db to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/958cfde2834a4f05b48b8d878879a3db 2023-03-28 20:54:12,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/342f16c47cf04ac7a655f4d4d8c3a498 2023-03-28 20:54:12,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/8644b97af5d34425a76cd4f19562459c 2023-03-28 20:54:12,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0b0e902ae7914c4ba8748b282ff6e951 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/0b0e902ae7914c4ba8748b282ff6e951 2023-03-28 20:54:12,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/ad0555f216804f2e80a89d7774db9718 2023-03-28 20:54:12,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/81925fde7e8b496f81fab6b22f4111e7 2023-03-28 20:54:12,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/c79e817ba8f248168a244c77d898a156 2023-03-28 20:54:12,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7da748b834044aa4a1555d183f888f76, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/b35dca0f42c64defa8b9a30dbb3a2f0b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/496ba064aa674c46adff6b37b3892d6c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/270cc39210d745269893cfa30851952c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/5b79a694de5b414e94469c7852d7351a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2459ea5c9ff04b39b1637e3eebe69ec4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/ab6c13c423504c9caeb76ffc5f90be5f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8603a441fcab47ca8dcaf1280fb7b8f8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/eb4e0d650e0d44ba941c64ebc88d17a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3] to archive 2023-03-28 20:54:12,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:12,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2ac3946f07d04ffa90f49ad3ff686a88 2023-03-28 20:54:12,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e86c45bf9c51429099cd75894ae0d652 2023-03-28 20:54:12,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7da748b834044aa4a1555d183f888f76 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7da748b834044aa4a1555d183f888f76 2023-03-28 20:54:12,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8d9165fa6e7248fa838cc2dde6d6f4a3 2023-03-28 20:54:12,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/a5e48a20b3924cd1be98f0999aa4f20a 2023-03-28 20:54:12,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/bd7bd90c8ae34784ad99b49c35c5fa1b 2023-03-28 20:54:12,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/b35dca0f42c64defa8b9a30dbb3a2f0b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/b35dca0f42c64defa8b9a30dbb3a2f0b 2023-03-28 20:54:12,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f7e73b2e424f4bdcb41d3e2076f96a37 2023-03-28 20:54:12,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fb9d4079b84d4aa58ce791cd6b2b02ca 2023-03-28 20:54:12,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/147cbcb148b7464982cea2f8fa2dcec6 2023-03-28 20:54:12,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/496ba064aa674c46adff6b37b3892d6c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/496ba064aa674c46adff6b37b3892d6c 2023-03-28 20:54:12,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4b0f17fc813041e7bfa87b31a30110f2 2023-03-28 20:54:12,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/4dcfaf69dcc748a0b1f54873f17dcda5 2023-03-28 20:54:12,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/270cc39210d745269893cfa30851952c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/270cc39210d745269893cfa30851952c 2023-03-28 20:54:12,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/14698d37869d4fc88f5d1ced02261402 2023-03-28 20:54:12,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/871a06864d2f448d9cce1211b46c4606 2023-03-28 20:54:12,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c93b72e6515e44a7a6b400b4d06adb47 2023-03-28 20:54:12,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/78e71b69bf7a4de9ab2ad8bbf6130e63 2023-03-28 20:54:12,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/e96caf793bf541639343145171d1ce7e 2023-03-28 20:54:12,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/87dcb5eb6d0c49ba9076bf2bc9e90b06 2023-03-28 20:54:12,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/f79a99a2cb2f4505a5d3012f94cd5ff9 2023-03-28 20:54:12,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/38547ea03f0b4c39915f91e2ed18bf3a 2023-03-28 20:54:12,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/5b79a694de5b414e94469c7852d7351a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/5b79a694de5b414e94469c7852d7351a 2023-03-28 20:54:12,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/6a1fea0f7ac74a0ab9cefcfd0c46f977 2023-03-28 20:54:12,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/446121b3b4004b8bb910d26049265a6b 2023-03-28 20:54:12,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2459ea5c9ff04b39b1637e3eebe69ec4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/2459ea5c9ff04b39b1637e3eebe69ec4 2023-03-28 20:54:12,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8a661a3f2b8544c08cb92e1728a15a04 2023-03-28 20:54:12,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/dfe508823a2447829f2c5e891107f486 2023-03-28 20:54:12,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/ab6c13c423504c9caeb76ffc5f90be5f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/ab6c13c423504c9caeb76ffc5f90be5f 2023-03-28 20:54:12,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cbd96bf2ac3b444dbe6588871e34d610 2023-03-28 20:54:12,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/c878b60a2bcc492799ad4c9ff5cb463c 2023-03-28 20:54:12,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cb6584f9ba2a42c192c08f521be385aa 2023-03-28 20:54:12,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8603a441fcab47ca8dcaf1280fb7b8f8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/8603a441fcab47ca8dcaf1280fb7b8f8 2023-03-28 20:54:12,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/584c91a9b2be4347bf2f53332a9317dc 2023-03-28 20:54:12,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/cde195ba14e2418bacd0c3e76456cf74 2023-03-28 20:54:12,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/eb4e0d650e0d44ba941c64ebc88d17a5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/eb4e0d650e0d44ba941c64ebc88d17a5 2023-03-28 20:54:12,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7b855c0129e742fc82f331d717c3da32 2023-03-28 20:54:12,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/33fd1deac4ca4d0e9918053245ec3242 2023-03-28 20:54:12,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/7f67247a6385414b9a9f4e6953b3fdb3 2023-03-28 20:54:12,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/fc411034a6c74af18c8ca8998a82b56b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b3857160bf67499597d63f113573e2cd, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aab71f48741d4691b3ce348b3363bb90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c51d222c0d2342fb95683d6fb85f2ac0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/e0a4ea1c64134efc936dd2ba98113d0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/02a797bf329e42629dd1b224e999aae8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d8dd5cc9ae6449288ffdaa5fa9e408af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/720437cde0f74bde89e591567ca68469, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169] to archive 2023-03-28 20:54:12,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:12,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d15ee1c054504de0b1edb7f63cd66be5 2023-03-28 20:54:12,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/863472eafeea41e8ab6278c76f6a2250 2023-03-28 20:54:12,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/fc411034a6c74af18c8ca8998a82b56b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/fc411034a6c74af18c8ca8998a82b56b 2023-03-28 20:54:12,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/df202e1913c34c1db53c9642241b18a6 2023-03-28 20:54:12,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff09de3e47324036829a172ed758a29b 2023-03-28 20:54:12,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cde56e12380466b99fa27c63cf3abd7 2023-03-28 20:54:12,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b3857160bf67499597d63f113573e2cd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b3857160bf67499597d63f113573e2cd 2023-03-28 20:54:12,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f660a7f9672c473c99581fd51635aa9d 2023-03-28 20:54:12,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/0a4f54a237ca471cbdf43fd6be9c58ed 2023-03-28 20:54:12,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/91591e4ba57b4e7597275b437631cd52 2023-03-28 20:54:12,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aab71f48741d4691b3ce348b3363bb90 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aab71f48741d4691b3ce348b3363bb90 2023-03-28 20:54:12,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/75b157a409ac4fff9a8a62e63b32875f 2023-03-28 20:54:12,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/903518400b4e4892bdc13ff0ca2a6961 2023-03-28 20:54:12,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c51d222c0d2342fb95683d6fb85f2ac0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c51d222c0d2342fb95683d6fb85f2ac0 2023-03-28 20:54:12,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/83d54fd5b4da4b42966b80db65a49bb3 2023-03-28 20:54:12,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/dd948e74c24d4eb78658cc9aa743db21 2023-03-28 20:54:12,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/5642388daf3c44d09bf46076d140a753 2023-03-28 20:54:12,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/67ca5113b76a4caf81cb49b23232ebe8 2023-03-28 20:54:12,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ff1ddbc858764ece983af3327e4d5613 2023-03-28 20:54:12,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1cbc1d4c73ca416db976d7f57666d87c 2023-03-28 20:54:12,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b2a070d718e94087b3c5efab5e8d5ecc 2023-03-28 20:54:12,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/ccbdce7177b14b0fafc65bbdebed5561 2023-03-28 20:54:12,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/e0a4ea1c64134efc936dd2ba98113d0a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/e0a4ea1c64134efc936dd2ba98113d0a 2023-03-28 20:54:12,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d3fbb0b433c2414488b2f3423bc34019 2023-03-28 20:54:12,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/1ddb77c9bb3249678f2a2c06d8f53984 2023-03-28 20:54:12,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/02a797bf329e42629dd1b224e999aae8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/02a797bf329e42629dd1b224e999aae8 2023-03-28 20:54:12,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/4a9dfcfa69444d458413324c73566a4f 2023-03-28 20:54:12,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/7277e88225a74cbf97a3f9d78dc69cd6 2023-03-28 20:54:12,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c325c1eed7bc4882952c7cdc1917f75a 2023-03-28 20:54:12,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/aa1b6888830247e89318796d45da663c 2023-03-28 20:54:12,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/a948b7dc76424cc8a5c5d641adc1563a 2023-03-28 20:54:12,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d8dd5cc9ae6449288ffdaa5fa9e408af to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/d8dd5cc9ae6449288ffdaa5fa9e408af 2023-03-28 20:54:12,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/60a2a7d6a08e4c70a8213d6527a90f11 2023-03-28 20:54:12,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/63c8810b56b844b0b47f80d5dd8cb2de 2023-03-28 20:54:12,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/720437cde0f74bde89e591567ca68469 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/720437cde0f74bde89e591567ca68469 2023-03-28 20:54:12,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/f8f3317cc4534e11960b180cb7a3edb2 2023-03-28 20:54:12,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/b7499d878d5845cca0e8e4c2b662e93b 2023-03-28 20:54:12,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/563c2ddb793d485985572e3ee1bf8169 2023-03-28 20:54:12,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/recovered.edits/669.seqid, newMaxSeqId=669, maxSeqId=1 2023-03-28 20:54:12,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef. 2023-03-28 20:54:12,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 59dd79967ef9e13a346f2f7f11b173ef: 2023-03-28 20:54:12,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:12,898 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=59dd79967ef9e13a346f2f7f11b173ef, regionState=CLOSED 2023-03-28 20:54:12,898 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036852898"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036852898"}]},"ts":"1680036852898"} 2023-03-28 20:54:12,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-28 20:54:12,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 59dd79967ef9e13a346f2f7f11b173ef, server=jenkins-hbase4.apache.org,42693,1680036815956 in 1.6360 sec 2023-03-28 20:54:12,902 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-28 20:54:12,902 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=59dd79967ef9e13a346f2f7f11b173ef, UNASSIGN in 1.6410 sec 2023-03-28 20:54:12,903 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036852903"}]},"ts":"1680036852903"} 2023-03-28 20:54:12,904 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:54:12,905 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:54:12,907 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7420 sec 2023-03-28 20:54:13,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-28 20:54:13,276 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-28 20:54:13,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:54:13,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,289 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,291 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-28 20:54:13,295 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:13,299 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/recovered.edits] 2023-03-28 20:54:13,302 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/04f9fe2a96ce415d8d883f07a74747f6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/04f9fe2a96ce415d8d883f07a74747f6 2023-03-28 20:54:13,303 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f31290739ceb47a89cdca7477e06e55a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/A/f31290739ceb47a89cdca7477e06e55a 2023-03-28 20:54:13,306 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/63d42d21ee2a4479892fa6e06074b3ab to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/63d42d21ee2a4479892fa6e06074b3ab 2023-03-28 20:54:13,307 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fbb56bf9f0e045a48d049f5038147ad6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/B/fbb56bf9f0e045a48d049f5038147ad6 2023-03-28 20:54:13,310 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/3750f2ebef5e4bd0b4952e349eb5247b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/3750f2ebef5e4bd0b4952e349eb5247b 2023-03-28 20:54:13,311 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c4aefb719308410dbe318829184f6264 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/C/c4aefb719308410dbe318829184f6264 2023-03-28 20:54:13,314 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/recovered.edits/669.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef/recovered.edits/669.seqid 2023-03-28 20:54:13,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/59dd79967ef9e13a346f2f7f11b173ef 2023-03-28 20:54:13,315 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:54:13,317 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,338 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:54:13,381 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:54:13,382 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,382 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:54:13,382 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036853382"}]},"ts":"9223372036854775807"} 2023-03-28 20:54:13,385 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:54:13,385 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 59dd79967ef9e13a346f2f7f11b173ef, NAME => 'TestAcidGuarantees,,1680036819794.59dd79967ef9e13a346f2f7f11b173ef.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:54:13,385 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:54:13,385 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036853385"}]},"ts":"9223372036854775807"} 2023-03-28 20:54:13,387 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:54:13,390 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,391 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 107 msec 2023-03-28 20:54:13,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-28 20:54:13,394 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-28 20:54:13,410 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:42693-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x191d92e5-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_702528059_15 at /127.0.0.1:53904 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x191d92e5-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x191d92e5-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x191d92e5-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=551 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=368 (was 416), ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=6908 (was 7535) 2023-03-28 20:54:13,422 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=368, ProcessCount=173, AvailableMemoryMB=6907 2023-03-28 20:54:13,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:54:13,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:54:13,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:13,426 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:54:13,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-28 20:54:13,427 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:54:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:13,429 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:13,429 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c empty. 2023-03-28 20:54:13,430 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:13,430 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:54:13,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:13,838 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:54:13,840 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 220e271bfe98d8e1f138571bcf50284c, NAME => 'TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:54:14,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 220e271bfe98d8e1f138571bcf50284c, disabling compactions & flushes 2023-03-28 20:54:14,248 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. after waiting 0 ms 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,248 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:14,250 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:54:14,252 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036854251"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036854251"}]},"ts":"1680036854251"} 2023-03-28 20:54:14,253 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:54:14,254 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:54:14,254 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036854254"}]},"ts":"1680036854254"} 2023-03-28 20:54:14,255 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:54:14,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, ASSIGN}] 2023-03-28 20:54:14,260 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, ASSIGN 2023-03-28 20:54:14,260 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:54:14,411 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:14,412 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036854411"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036854411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036854411"}]},"ts":"1680036854411"} 2023-03-28 20:54:14,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:14,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:14,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 220e271bfe98d8e1f138571bcf50284c, NAME => 'TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:54:14,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:54:14,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,571 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,572 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A 2023-03-28 20:54:14,572 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A 2023-03-28 20:54:14,572 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:14,573 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName A 2023-03-28 20:54:14,573 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:14,573 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,574 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B 2023-03-28 20:54:14,574 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B 2023-03-28 20:54:14,575 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:14,575 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName B 2023-03-28 20:54:14,575 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:14,575 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,576 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C 2023-03-28 20:54:14,576 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C 2023-03-28 20:54:14,577 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:14,577 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName C 2023-03-28 20:54:14,577 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:14,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:54:14,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:14,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:54:14,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 220e271bfe98d8e1f138571bcf50284c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66948470, jitterRate=-0.002390056848526001}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:54:14,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:14,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., pid=18, masterSystemTime=1680036854565 2023-03-28 20:54:14,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:14,587 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:14,587 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036854587"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036854587"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036854587"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036854587"}]},"ts":"1680036854587"} 2023-03-28 20:54:14,590 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-28 20:54:14,590 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 in 176 msec 2023-03-28 20:54:14,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-28 20:54:14,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, ASSIGN in 332 msec 2023-03-28 20:54:14,592 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:54:14,593 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036854593"}]},"ts":"1680036854593"} 2023-03-28 20:54:14,594 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:54:14,596 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:54:14,597 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-03-28 20:54:15,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-28 20:54:15,533 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-28 20:54:15,534 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x58d22e5a to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:15,540 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4910cda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:15,541 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:15,545 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49576, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:15,546 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:54:15,548 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34488, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:54:15,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:54:15,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:54:15,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:15,974 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:54:15,976 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:54:15,976 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:54:15,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-28 20:54:15,993 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, REOPEN/MOVE}] 2023-03-28 20:54:15,994 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, REOPEN/MOVE 2023-03-28 20:54:15,995 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:15,995 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036855995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036855995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036855995"}]},"ts":"1680036855995"} 2023-03-28 20:54:15,997 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:16,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 220e271bfe98d8e1f138571bcf50284c, disabling compactions & flushes 2023-03-28 20:54:16,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. after waiting 0 ms 2023-03-28 20:54:16,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-28 20:54:16,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:16,155 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 220e271bfe98d8e1f138571bcf50284c to self. 2023-03-28 20:54:16,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,157 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=CLOSED 2023-03-28 20:54:16,157 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036856157"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036856157"}]},"ts":"1680036856157"} 2023-03-28 20:54:16,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-28 20:54:16,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 in 161 msec 2023-03-28 20:54:16,160 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=true 2023-03-28 20:54:16,310 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,310 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036856310"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036856310"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036856310"}]},"ts":"1680036856310"} 2023-03-28 20:54:16,312 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:16,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 220e271bfe98d8e1f138571bcf50284c, NAME => 'TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:54:16,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:54:16,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,470 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,471 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A 2023-03-28 20:54:16,471 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A 2023-03-28 20:54:16,471 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:16,474 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName A 2023-03-28 20:54:16,476 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:16,476 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,477 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B 2023-03-28 20:54:16,477 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B 2023-03-28 20:54:16,477 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:16,478 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName B 2023-03-28 20:54:16,478 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:16,478 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,479 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C 2023-03-28 20:54:16,479 DEBUG [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C 2023-03-28 20:54:16,479 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:16,480 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 220e271bfe98d8e1f138571bcf50284c columnFamilyName C 2023-03-28 20:54:16,480 INFO [StoreOpener-220e271bfe98d8e1f138571bcf50284c-1] regionserver.HStore(310): Store=220e271bfe98d8e1f138571bcf50284c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:16,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:54:16,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 220e271bfe98d8e1f138571bcf50284c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60128839, jitterRate=-0.10401047766208649}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:54:16,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:16,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., pid=23, masterSystemTime=1680036856464 2023-03-28 20:54:16,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,488 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,488 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036856488"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036856488"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036856488"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036856488"}]},"ts":"1680036856488"} 2023-03-28 20:54:16,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-28 20:54:16,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 in 178 msec 2023-03-28 20:54:16,492 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-28 20:54:16,492 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, REOPEN/MOVE in 498 msec 2023-03-28 20:54:16,495 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-28 20:54:16,495 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-03-28 20:54:16,497 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 938 msec 2023-03-28 20:54:16,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-28 20:54:16,505 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x542e33ab to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,510 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c508332, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,511 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x46078b1e to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,515 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7adf5e8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,515 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x28fcbe97 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,523 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52d98fed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,524 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,531 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,531 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x76aef981 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,535 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25441895, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,535 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,539 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,539 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x63d375f8 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,543 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2668d41d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,543 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1345b761 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,548 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b416c44, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,548 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1e0b0a9f to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:16,551 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@204b5880, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:16,558 DEBUG [hconnection-0x49b5ea7b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,558 DEBUG [hconnection-0x45329129-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,558 DEBUG [hconnection-0x176c4c2d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,558 DEBUG [hconnection-0x28a360da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,558 DEBUG [hconnection-0x116dc3e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,559 DEBUG [hconnection-0x363ff606-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,561 DEBUG [hconnection-0x281bd44f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,563 DEBUG [hconnection-0x3b0bae0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,563 DEBUG [hconnection-0x1a1e03e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:16,563 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49594, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:16,565 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49600, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,566 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,566 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,565 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49578, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,567 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49660, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,567 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49636, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,568 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,569 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49650, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:16,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:16,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:16,578 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:16,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:16,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:16,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:16,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:16,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:16,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:16,594 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,594 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:16,594 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:16,594 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:16,594 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:16,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,597 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,597 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:16,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272465c9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:16,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:16,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:16,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:16,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:16,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:16,604 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:16,604 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,604 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:16,604 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:16,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:16,607 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:16,607 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:16,607 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:16,607 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:16,607 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:16,609 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,609 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,609 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,615 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:16,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:16,615 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:16,615 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:16,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036916615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036916615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036916620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036916621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036916622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,654 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303287714e3fee57a42a8806316c9f1a972eb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287714e3fee57a42a8806316c9f1a972eb 2023-03-28 20:54:16,688 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/c5ac10000a794e548705ba2582246df9 2023-03-28 20:54:16,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/8ed4bbae18b84c81bb28ccdcd06c2a31 2023-03-28 20:54:16,715 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:16,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:16,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:16,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:16,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:16,724 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:16,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:16,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:16,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036916726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036916727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,732 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,732 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:16,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:16,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036916733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036916733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036916734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:16,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:16,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:16,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:16,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:16,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:16,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:16,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f83ca4c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:16,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:16,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:16,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:16,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:16,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:16,770 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:16,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:16,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:16,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:16,771 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:16,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,774 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,775 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:16,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:16,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,779 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:16,779 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,879 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:16,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:16,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:16,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:16,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:16,892 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:16,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:16,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:16,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:16,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:16,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:16,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:16,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:16,911 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,911 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:16,911 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:16,911 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,914 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654481ea[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:16,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:16,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,918 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:16,918 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:16,918 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:16,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:16,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:16,923 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:16,923 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:16,923 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:16,923 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:16,923 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:16,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:16,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:16,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:16,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036916934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036916934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:16,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036916936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:16,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036916937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:16,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036916936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:16,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:16,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:16,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:16,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:16,941 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:16,941 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:16,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:16,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:17,041 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:17,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:17,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:17,045 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:17,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:17,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:17,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:17,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,047 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:17,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:17,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:17,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:17,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:17,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:17,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:17,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:17,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:17,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:17,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:17,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:17,061 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:17,061 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625914f3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:17,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:17,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:17,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:17,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:17,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:17,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:17,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:17,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:17,069 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:17,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:17,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:17,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:17,069 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:17,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:17,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:17,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,088 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:17,088 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:17,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:17,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:17,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:17,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:17,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:17,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:17,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,090 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:17,090 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:17,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/d898cdbc4a37404db92e920831cf4936 2023-03-28 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/c5ac10000a794e548705ba2582246df9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9 2023-03-28 20:54:17,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9, entries=150, sequenceid=17, filesize=25.1 K 2023-03-28 20:54:17,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/8ed4bbae18b84c81bb28ccdcd06c2a31 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31 2023-03-28 20:54:17,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:54:17,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/d898cdbc4a37404db92e920831cf4936 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936 2023-03-28 20:54:17,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:54:17,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 220e271bfe98d8e1f138571bcf50284c in 575ms, sequenceid=17, compaction requested=false 2023-03-28 20:54:17,168 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-28 20:54:17,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:17,190 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:17,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:17,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:17,197 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:17,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:17,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:17,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:17,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:17,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:17,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:17,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:17,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:17,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:17,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:17,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:17,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:17,216 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:17,216 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:17,218 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:54:17,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:17,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:17,219 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:17,219 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:17,219 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:17,219 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:17,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:17,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036917246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036917245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036917246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036917249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036917249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036917350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036917350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036917351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036917350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036917353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036917552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036917553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036917553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036917554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036917555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,677 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328ad331ebd14da4c7a8fa100f70383c450 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ad331ebd14da4c7a8fa100f70383c450 2023-03-28 20:54:17,686 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/bfe1a7bac04f48afab763c3f0b64c5f3 2023-03-28 20:54:17,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036917855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036917857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036917857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036917857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:17,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:17,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036917859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,107 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/671db35c06204f4da6011303e5c0acfb 2023-03-28 20:54:18,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:18,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036918360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:18,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036918360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036918362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036918362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036918367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,439 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:54:18,534 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/e9d6cdb103674270a5dd35b25e809b90 2023-03-28 20:54:18,540 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/bfe1a7bac04f48afab763c3f0b64c5f3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3 2023-03-28 20:54:18,547 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3, entries=150, sequenceid=37, filesize=25.1 K 2023-03-28 20:54:18,547 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-28 20:54:18,548 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/671db35c06204f4da6011303e5c0acfb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb 2023-03-28 20:54:18,554 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb, entries=150, sequenceid=37, filesize=11.6 K 2023-03-28 20:54:18,555 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/e9d6cdb103674270a5dd35b25e809b90 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90 2023-03-28 20:54:18,560 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90, entries=150, sequenceid=37, filesize=11.6 K 2023-03-28 20:54:18,562 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 220e271bfe98d8e1f138571bcf50284c in 1344ms, sequenceid=37, compaction requested=false 2023-03-28 20:54:18,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:18,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:18,567 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:18,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d72f5c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:18,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,571 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:18,571 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,571 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:18,574 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:18,577 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:18,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:18,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:18,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:18,578 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:18,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:18,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:18,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:18,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:18,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:18,598 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:18,598 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:18,698 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:18,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:18,709 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:18,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:18,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:18,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:18,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:18,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:18,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:18,716 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:18,716 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:18,716 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:18,717 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:18,763 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032888a20aae29bf47e59b913d147f4253af to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032888a20aae29bf47e59b913d147f4253af 2023-03-28 20:54:18,781 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/0345a935d2744891aa46ec45811bb7ad 2023-03-28 20:54:18,800 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/f4a45a9114cb4794aa7cf613af30dc29 2023-03-28 20:54:18,817 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/94384d72d3c04334b77673a9982dcd7c 2023-03-28 20:54:18,823 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/0345a935d2744891aa46ec45811bb7ad as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad 2023-03-28 20:54:18,828 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad, entries=100, sequenceid=50, filesize=18.4 K 2023-03-28 20:54:18,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/f4a45a9114cb4794aa7cf613af30dc29 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29 2023-03-28 20:54:18,837 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29, entries=100, sequenceid=50, filesize=9.3 K 2023-03-28 20:54:18,838 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/94384d72d3c04334b77673a9982dcd7c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c 2023-03-28 20:54:18,846 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c, entries=100, sequenceid=50, filesize=9.3 K 2023-03-28 20:54:18,847 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 220e271bfe98d8e1f138571bcf50284c in 130ms, sequenceid=50, compaction requested=true 2023-03-28 20:54:18,847 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:18,847 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,848 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,849 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f75860[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,852 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:18,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:18,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:18,858 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:18,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:18,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:18,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:18,858 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:18,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:18,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:18,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,864 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:18,867 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:18,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:18,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,867 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:18,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:18,867 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:18,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:18,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:18,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:18,968 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:18,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:18,973 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:18,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:18,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:18,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:18,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,975 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:18,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:18,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:18,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:18,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:18,978 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:18,978 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:18,978 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:18,978 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:18,980 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,981 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,981 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567dd0a5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:18,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:18,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,985 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:18,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:18,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:18,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:18,990 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:18,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:18,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:18,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:18,990 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:19,013 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,013 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,013 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:19,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,019 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:19,019 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:19,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,120 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:19,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:19,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:19,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:19,124 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,126 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:19,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:19,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:19,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:19,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:19,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca9a815[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:19,134 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:19,134 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,135 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:19,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:19,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:19,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:19,142 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:19,146 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:19,146 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:19,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,150 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:19,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:19,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,153 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:19,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:19,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,154 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,154 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:19,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,254 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:19,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:19,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:19,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:19,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:19,261 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:19,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:19,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:19,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:19,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:19,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:19,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:19,268 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:19,268 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:19,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dbc513[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:19,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:19,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:19,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:19,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:19,274 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,274 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:19,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:19,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:19,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:19,278 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:19,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:19,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:19,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:19,278 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:19,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:19,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:19,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:19,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:19,292 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:19,292 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:19,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:19,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:19,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:19,392 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:19,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:19,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:19,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:19,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:19,399 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:19,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:19,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:19,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:19,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:19,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:19,403 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,403 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:19,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:19,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:19,405 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-28 20:54:19,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:19,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:19,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:19,407 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:19,407 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:19,407 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:19,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:19,432 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328ff2179239da5411380cd08b4396ef1bd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ff2179239da5411380cd08b4396ef1bd 2023-03-28 20:54:19,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036859450, max timestamp=1680036859454Num uniques 859; 2023-03-28 20:54:19,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036859450, max timestamp=1680036859454Num uniques 859; 2023-03-28 20:54:19,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036859450, max timestamp=1680036859454Num uniques 859; 2023-03-28 20:54:19,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:19,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:19,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:19,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036859472, max timestamp=1680036859479Num uniques 1068; 2023-03-28 20:54:19,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036859472, max timestamp=1680036859479Num uniques 1068; 2023-03-28 20:54:19,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-28 20:54:19,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036859472, max timestamp=1680036859479Num uniques 1068; 2023-03-28 20:54:19,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-28 20:54:19,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1680036859493, max timestamp=1680036859523Num uniques 1027; 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1680036859493, max timestamp=1680036859523Num uniques 1027; 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=19552, suffix heap size=134776,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=427 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=19552, suffix heap size=134776,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=427 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1680036859493, max timestamp=1680036859523Num uniques 1027; 2023-03-28 20:54:19,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=19552, suffix heap size=134776,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=427 2023-03-28 20:54:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036919542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036919544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036919544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036919545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036919547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036919652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036919653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036919654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036919654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036919654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036919855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036919855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036919856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:19,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036919856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036919856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:19,861 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=59, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/96083869ea02429fa1c4d50421de35a9 2023-03-28 20:54:19,872 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/014e001552bf420f93035729e708cec1 2023-03-28 20:54:20,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036920158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036920159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036920159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036920159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036920160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,282 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/cdc248d03ecd45238b7a9458062aa7f6 2023-03-28 20:54:20,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/96083869ea02429fa1c4d50421de35a9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9 2023-03-28 20:54:20,292 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9, entries=150, sequenceid=59, filesize=25.1 K 2023-03-28 20:54:20,293 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/014e001552bf420f93035729e708cec1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1 2023-03-28 20:54:20,299 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1, entries=100, sequenceid=59, filesize=9.3 K 2023-03-28 20:54:20,300 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/cdc248d03ecd45238b7a9458062aa7f6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6 2023-03-28 20:54:20,305 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6, entries=100, sequenceid=59, filesize=9.3 K 2023-03-28 20:54:20,306 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for 220e271bfe98d8e1f138571bcf50284c in 901ms, sequenceid=59, compaction requested=true 2023-03-28 20:54:20,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:20,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:20,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec2bd50[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:20,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:20,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,321 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:20,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:20,322 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:20,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:20,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:20,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:20,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:20,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:20,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,330 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,334 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:20,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,334 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:20,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,334 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:20,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,435 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:20,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:20,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:20,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:20,438 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:20,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:20,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:20,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:20,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:20,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:20,443 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,443 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:20,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=167.72 KB heapSize=459.47 KB 2023-03-28 20:54:20,443 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:20,468 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032802219e0165494467af1974aab7e00886 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032802219e0165494467af1974aab7e00886 2023-03-28 20:54:20,478 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8e4e6ddec3ad4fcdbe4b70006c4344bf 2023-03-28 20:54:20,494 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/d449d7ef15844176a0b3a42bb735c794 2023-03-28 20:54:20,503 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/f7d86a007b8843d2a7f2afd092be9c39 2023-03-28 20:54:20,507 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8e4e6ddec3ad4fcdbe4b70006c4344bf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf 2023-03-28 20:54:20,511 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf, entries=150, sequenceid=130, filesize=25.2 K 2023-03-28 20:54:20,512 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/d449d7ef15844176a0b3a42bb735c794 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794 2023-03-28 20:54:20,516 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794, entries=150, sequenceid=130, filesize=11.7 K 2023-03-28 20:54:20,517 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/f7d86a007b8843d2a7f2afd092be9c39 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39 2023-03-28 20:54:20,521 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39, entries=150, sequenceid=130, filesize=11.7 K 2023-03-28 20:54:20,522 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~459.42 KB/470448, currentSize=0 B/0 for 220e271bfe98d8e1f138571bcf50284c in 79ms, sequenceid=130, compaction requested=true 2023-03-28 20:54:20,523 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:20,523 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:20,525 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3682acd5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:20,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:20,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:20,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:20,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:20,529 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,529 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:20,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:20,536 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:20,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:20,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:20,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:20,536 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:20,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,541 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:20,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:20,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:20,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,547 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:20,548 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:20,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,648 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:20,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:20,655 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:20,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:20,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:20,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,657 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:20,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:20,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:20,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:20,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:20,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:20,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:20,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a4310[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:20,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:20,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:20,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:20,689 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:20,692 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:20,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:20,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:20,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:20,693 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:20,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:20,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:20,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,701 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:20,701 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:20,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:20,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:20,749 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303284a15ef66fa654ee9af8dc3415071cd4e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284a15ef66fa654ee9af8dc3415071cd4e 2023-03-28 20:54:20,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036920776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036860756, max timestamp=1680036860780Num uniques 859; 2023-03-28 20:54:20,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:20,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036920778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036860756, max timestamp=1680036860780Num uniques 859; 2023-03-28 20:54:20,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:20,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036860756, max timestamp=1680036860780Num uniques 859; 2023-03-28 20:54:20,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:20,801 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:20,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:20,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:20,807 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:20,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:20,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:20,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,810 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:20,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:20,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:20,815 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,815 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:20,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:20,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:20,818 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:20,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad4edb5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:20,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:20,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:20,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:20,824 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036920818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036920825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036920827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:20,830 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:20,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:20,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:20,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:20,831 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:20,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:20,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,839 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,839 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:20,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036920887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036920890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036920928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036920929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:20,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036920930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,940 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:20,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:20,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:20,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:20,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:20,943 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:20,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:20,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:20,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,945 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,945 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:20,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:20,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:20,947 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,947 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:20,948 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:20,948 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:20,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6167f782[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:20,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:20,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:20,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:20,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:20,963 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:20,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:20,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:20,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:20,963 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:20,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:20,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:20,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:20,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:20,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:20,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:20,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:20,972 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:20,972 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:20,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:20,973 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,073 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,076 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,077 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,077 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,082 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,082 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,082 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,082 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a00b7f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,089 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,091 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036921092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,095 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,095 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,101 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,101 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036921096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036921131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036921134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036921137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,160 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/ea0343d5b0ef4e02bee27190c6260026 2023-03-28 20:54:21,201 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,205 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,205 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,205 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,207 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,207 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,209 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,212 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,212 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,212 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,212 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,214 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173792f8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,220 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,223 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,223 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,226 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,226 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,226 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,229 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,233 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,233 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,333 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,344 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,346 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,351 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a536cee[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,356 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,358 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,360 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,360 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,362 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,362 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,362 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,366 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,366 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,366 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036921396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036921405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036921440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036921440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036921440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,466 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,471 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,486 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,486 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,486 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145649c1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,489 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,499 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,499 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,501 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,501 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,502 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,504 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,507 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,508 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,508 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/90c1415c669043adbe23bdb246ed805d 2023-03-28 20:54:21,608 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,628 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,630 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,630 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,640 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,640 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,640 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,640 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,644 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566f43c7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,650 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,653 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,653 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,653 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,653 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,653 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,655 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,655 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,656 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,661 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,666 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,666 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,767 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,771 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,772 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,774 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,774 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,774 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,774 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d64aba0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,781 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,781 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,783 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,786 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,786 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,788 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,790 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,794 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,794 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,894 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:21,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:21,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:21,898 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:21,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:21,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:21,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:21,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:21,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:21,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:21,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:21,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:21,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:21,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036921901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,925 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c018cfe[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:21,929 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:21,929 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036921927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,930 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,930 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:21,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:21,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:21,935 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:21,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:21,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:21,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:21,935 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:21,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:21,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:21,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:21,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:21,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:21,945 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:21,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:21,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:21,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:21,945 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:21,945 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:21,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:21,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036921944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036921945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:21,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036921946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/9fadc091d6774bcb82fe3b9a976bf7eb 2023-03-28 20:54:22,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/ea0343d5b0ef4e02bee27190c6260026 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026 2023-03-28 20:54:22,045 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:22,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:22,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026, entries=400, sequenceid=142, filesize=59.4 K 2023-03-28 20:54:22,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:22,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/90c1415c669043adbe23bdb246ed805d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d 2023-03-28 20:54:22,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:22,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:22,052 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:22,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:22,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:22,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:22,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:22,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:22,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:22,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:22,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:22,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:22,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:54:22,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:22,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/9fadc091d6774bcb82fe3b9a976bf7eb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb 2023-03-28 20:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:22,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:54:22,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 220e271bfe98d8e1f138571bcf50284c in 1354ms, sequenceid=142, compaction requested=true 2023-03-28 20:54:22,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:22,064 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:54:22,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:22,065 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:54:22,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:22,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:22,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:22,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:22,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:22,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:22,065 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:22,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:22,067 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 182682 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:54:22,067 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:22,067 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:22,067 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=178.4 K 2023-03-28 20:54:22,072 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c5ac10000a794e548705ba2582246df9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036856587 2023-03-28 20:54:22,073 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66828 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:54:22,073 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:22,073 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:22,074 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=65.3 K 2023-03-28 20:54:22,074 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting bfe1a7bac04f48afab763c3f0b64c5f3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036856602 2023-03-28 20:54:22,074 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8ed4bbae18b84c81bb28ccdcd06c2a31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036856587 2023-03-28 20:54:22,075 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0345a935d2744891aa46ec45811bb7ad, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036857246 2023-03-28 20:54:22,075 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 671db35c06204f4da6011303e5c0acfb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036856602 2023-03-28 20:54:22,075 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 96083869ea02429fa1c4d50421de35a9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680036859402 2023-03-28 20:54:22,076 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f4a45a9114cb4794aa7cf613af30dc29, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036857246 2023-03-28 20:54:22,077 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8e4e6ddec3ad4fcdbe4b70006c4344bf, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680036859539 2023-03-28 20:54:22,077 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 014e001552bf420f93035729e708cec1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680036859402 2023-03-28 20:54:22,078 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ea0343d5b0ef4e02bee27190c6260026, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860687 2023-03-28 20:54:22,078 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d449d7ef15844176a0b3a42bb735c794, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680036859539 2023-03-28 20:54:22,079 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 90c1415c669043adbe23bdb246ed805d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860693 2023-03-28 20:54:22,112 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#141 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:22,143 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#142 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:22,195 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/7ce45558e2aa4da598dd7bf4c555446d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7ce45558e2aa4da598dd7bf4c555446d 2023-03-28 20:54:22,200 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into 7ce45558e2aa4da598dd7bf4c555446d(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:22,201 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:22,201 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=10, startTime=1680036862064; duration=0sec 2023-03-28 20:54:22,201 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:22,201 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:54:22,204 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66828 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:54:22,204 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:22,204 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:22,205 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=65.3 K 2023-03-28 20:54:22,205 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d898cdbc4a37404db92e920831cf4936, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036856587 2023-03-28 20:54:22,205 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e9d6cdb103674270a5dd35b25e809b90, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036856602 2023-03-28 20:54:22,206 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 94384d72d3c04334b77673a9982dcd7c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036857246 2023-03-28 20:54:22,206 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cdc248d03ecd45238b7a9458062aa7f6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680036859402 2023-03-28 20:54:22,207 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f7d86a007b8843d2a7f2afd092be9c39, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680036859539 2023-03-28 20:54:22,208 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 9fadc091d6774bcb82fe3b9a976bf7eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860693 2023-03-28 20:54:22,222 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:22,245 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/dab8ddc60b7c4a83a72765906f1634ac as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/dab8ddc60b7c4a83a72765906f1634ac 2023-03-28 20:54:22,251 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into dab8ddc60b7c4a83a72765906f1634ac(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:22,251 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:22,251 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=10, startTime=1680036862065; duration=0sec 2023-03-28 20:54:22,251 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:22,546 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032844eae8b4a84d4da9a98207180828f7d5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032844eae8b4a84d4da9a98207180828f7d5 2023-03-28 20:54:22,559 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=177, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/3dfbcb6f7b9b4003a922c1d6a75b74d4 2023-03-28 20:54:22,561 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/b8def24c8b3c4d94b8fcefcd96f6ef8d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/b8def24c8b3c4d94b8fcefcd96f6ef8d 2023-03-28 20:54:22,568 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into b8def24c8b3c4d94b8fcefcd96f6ef8d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:22,569 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:22,569 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=10, startTime=1680036862065; duration=0sec 2023-03-28 20:54:22,569 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:22,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:22,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:22,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:22,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036922950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:22,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036922952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036922953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036922953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036922954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:22,979 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/4b4a7598589c4369980043d8bf037450 2023-03-28 20:54:22,991 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/c768e5fffd3c42659a12ad5a97ec893b 2023-03-28 20:54:22,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/3dfbcb6f7b9b4003a922c1d6a75b74d4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4 2023-03-28 20:54:23,000 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4, entries=150, sequenceid=177, filesize=25.3 K 2023-03-28 20:54:23,000 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/4b4a7598589c4369980043d8bf037450 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450 2023-03-28 20:54:23,004 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450, entries=150, sequenceid=177, filesize=11.8 K 2023-03-28 20:54:23,005 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/c768e5fffd3c42659a12ad5a97ec893b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b 2023-03-28 20:54:23,009 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b, entries=150, sequenceid=177, filesize=11.8 K 2023-03-28 20:54:23,010 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 220e271bfe98d8e1f138571bcf50284c in 945ms, sequenceid=177, compaction requested=false 2023-03-28 20:54:23,010 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:23,010 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,012 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abfd0fe[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,017 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,017 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,017 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,023 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,023 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,025 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,025 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,025 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,031 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,031 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,032 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,032 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,032 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,032 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,032 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:23,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:23,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:23,070 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328444be8ae14414aaa9c9d18db0ebff599 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328444be8ae14414aaa9c9d18db0ebff599 2023-03-28 20:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036923102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036923103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036923107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,132 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,135 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,137 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,139 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,139 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,139 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,139 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2662cb15[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,147 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,151 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,151 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,154 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,155 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,158 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,159 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,159 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036923208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036923210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036923211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,259 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,262 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e035dd2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,284 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,285 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,296 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,296 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,396 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,400 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,404 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,404 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,404 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,406 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,408 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,408 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e454890[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036923413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036923413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036923414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,422 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,422 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,424 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,424 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,424 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,429 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,430 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,479 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/26d19e186ac3458993c1ec47497ba52b 2023-03-28 20:54:23,530 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,533 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,534 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,535 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,536 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,537 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,537 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a095b86[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,546 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,546 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,552 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,552 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,652 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,655 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,664 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,665 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,665 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0307b1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,668 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,668 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,669 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,675 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,675 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,678 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,678 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,679 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,680 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,687 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,687 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:23,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036923716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036923717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:23,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036923717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,787 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,792 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,793 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4a5dd5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,831 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,834 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,834 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,851 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,851 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:23,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/ce8e20f7328745afac7a6d950d18bdf1 2023-03-28 20:54:23,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/9967093c989343ecaae1299f83a8b30c 2023-03-28 20:54:23,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/26d19e186ac3458993c1ec47497ba52b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b 2023-03-28 20:54:23,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b, entries=200, sequenceid=193, filesize=32.1 K 2023-03-28 20:54:23,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/ce8e20f7328745afac7a6d950d18bdf1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1 2023-03-28 20:54:23,951 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:23,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:23,957 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:23,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1, entries=150, sequenceid=193, filesize=11.8 K 2023-03-28 20:54:23,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:23,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:23,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/9967093c989343ecaae1299f83a8b30c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c 2023-03-28 20:54:23,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:23,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:23,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:23,961 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,961 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:23,962 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:23,962 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c, entries=150, sequenceid=193, filesize=11.8 K 2023-03-28 20:54:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 220e271bfe98d8e1f138571bcf50284c in 910ms, sequenceid=193, compaction requested=true 2023-03-28 20:54:23,967 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:23,967 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:23,967 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:23,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:23,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6876e1a2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:23,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:23,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,968 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:23,968 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:23,968 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,968 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/b8def24c8b3c4d94b8fcefcd96f6ef8d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=35.5 K 2023-03-28 20:54:23,969 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:23,969 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:23,969 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b8def24c8b3c4d94b8fcefcd96f6ef8d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860693 2023-03-28 20:54:23,969 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:23,969 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,969 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7ce45558e2aa4da598dd7bf4c555446d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=82.9 K 2023-03-28 20:54:23,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:23,970 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4b4a7598589c4369980043d8bf037450, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680036860802 2023-03-28 20:54:23,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:23,970 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7ce45558e2aa4da598dd7bf4c555446d, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860693 2023-03-28 20:54:23,970 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ce8e20f7328745afac7a6d950d18bdf1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:23,970 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 3dfbcb6f7b9b4003a922c1d6a75b74d4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680036860802 2023-03-28 20:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:23,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,971 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 26d19e186ac3458993c1ec47497ba52b, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:23,978 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:23,978 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:23,980 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:23,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:23,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:23,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:23,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:23,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:23,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:23,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:23,989 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:23,989 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:23,994 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:23,998 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#149 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:24,031 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/5a7d28da8d1046ccb2433412d1aa4c79 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/5a7d28da8d1046ccb2433412d1aa4c79 2023-03-28 20:54:24,037 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into 5a7d28da8d1046ccb2433412d1aa4c79(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:24,038 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:24,038 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=13, startTime=1680036863967; duration=0sec 2023-03-28 20:54:24,038 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:24,038 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:24,040 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:24,040 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:24,040 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:24,040 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/dab8ddc60b7c4a83a72765906f1634ac, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=35.5 K 2023-03-28 20:54:24,041 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting dab8ddc60b7c4a83a72765906f1634ac, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036860693 2023-03-28 20:54:24,041 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c768e5fffd3c42659a12ad5a97ec893b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680036860802 2023-03-28 20:54:24,041 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 9967093c989343ecaae1299f83a8b30c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:24,050 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#151 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:24,057 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/7efcf89b71c546cda3657d3812ca8576 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7efcf89b71c546cda3657d3812ca8576 2023-03-28 20:54:24,062 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/345706e1daa74414a8e772da7e797f1f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/345706e1daa74414a8e772da7e797f1f 2023-03-28 20:54:24,068 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into 7efcf89b71c546cda3657d3812ca8576(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:24,068 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:24,068 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=13, startTime=1680036863967; duration=0sec 2023-03-28 20:54:24,068 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:24,076 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into 345706e1daa74414a8e772da7e797f1f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:24,076 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:24,077 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=13, startTime=1680036863967; duration=0sec 2023-03-28 20:54:24,077 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:24,089 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:24,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:24,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:24,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:24,094 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:24,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:24,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:24,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:24,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,096 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:24,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:24,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:24,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:24,100 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:24,100 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:24,100 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,133 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303282a509616bea14a6f9cda4dbe457eb0d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282a509616bea14a6f9cda4dbe457eb0d0 2023-03-28 20:54:24,148 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/06a1775c0ae34bf89350e8b3096b22a6 2023-03-28 20:54:24,178 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/16be89d9b38247d099003b9c59caf9d0 2023-03-28 20:54:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:24,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:24,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036924229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036924230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036924232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036924333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036924333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036924335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036924537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036924537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036924538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,592 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/fcb0751feb834c33be91cdc3710e164c 2023-03-28 20:54:24,597 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/06a1775c0ae34bf89350e8b3096b22a6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6 2023-03-28 20:54:24,601 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6, entries=150, sequenceid=215, filesize=25.3 K 2023-03-28 20:54:24,602 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/16be89d9b38247d099003b9c59caf9d0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0 2023-03-28 20:54:24,606 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0, entries=150, sequenceid=215, filesize=11.8 K 2023-03-28 20:54:24,607 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/fcb0751feb834c33be91cdc3710e164c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c 2023-03-28 20:54:24,611 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c, entries=150, sequenceid=215, filesize=11.8 K 2023-03-28 20:54:24,611 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 220e271bfe98d8e1f138571bcf50284c in 511ms, sequenceid=215, compaction requested=false 2023-03-28 20:54:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:24,613 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f41516[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:24,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:24,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:24,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:24,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:24,621 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:24,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:24,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:24,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:24,622 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:24,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,628 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:24,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:24,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:24,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:24,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:24,628 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,628 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:24,629 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:24,629 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:24,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:24,729 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:24,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:24,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:24,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:24,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:24,734 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:24,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:24,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:24,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:24,737 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,737 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,738 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:24,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:24,738 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:24,740 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,740 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:24,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:24,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:24,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:24,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:24,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:24,745 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:24,745 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:24,745 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:24,745 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:24,746 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:24,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:24,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036924875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036924875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036924876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036924976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,978 DEBUG [Thread-929] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:24,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036924979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036924979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036924979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036924979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:24,982 DEBUG [Thread-923] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:25,172 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303287cd3e7fc780d4ac6b1f39ce7aad5df89 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287cd3e7fc780d4ac6b1f39ce7aad5df89 2023-03-28 20:54:25,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036925182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036925182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,183 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/77feea442fc84794a146483bd963ae0a 2023-03-28 20:54:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036925182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036925483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036925485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036925486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,618 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/69389325d1e54ce5a6550d55c7a7ddf9 2023-03-28 20:54:25,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036925988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036925989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036925990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,039 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/d35046e9cf114f9ab2194223b925b848 2023-03-28 20:54:26,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/77feea442fc84794a146483bd963ae0a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a 2023-03-28 20:54:26,049 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a, entries=150, sequenceid=228, filesize=25.3 K 2023-03-28 20:54:26,051 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/69389325d1e54ce5a6550d55c7a7ddf9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9 2023-03-28 20:54:26,064 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9, entries=150, sequenceid=228, filesize=11.8 K 2023-03-28 20:54:26,066 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/d35046e9cf114f9ab2194223b925b848 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848 2023-03-28 20:54:26,072 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848, entries=150, sequenceid=228, filesize=11.8 K 2023-03-28 20:54:26,073 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 220e271bfe98d8e1f138571bcf50284c in 1327ms, sequenceid=228, compaction requested=true 2023-03-28 20:54:26,073 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:26,073 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:26,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ed2b55[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:26,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:26,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,083 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:26,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:26,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:26,089 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:26,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:26,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:26,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:26,089 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:26,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:26,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:26,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:26,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:26,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,099 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:26,099 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:26,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:26,199 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:26,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:26,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:26,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:26,202 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:26,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:26,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:26,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:26,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:26,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:26,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:26,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:26,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:26,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:26,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:26,206 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:26,206 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:26,206 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:26,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:26,636 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328458578387d944afa910aff6545c1df1c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328458578387d944afa910aff6545c1df1c 2023-03-28 20:54:26,649 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/3340111a5ea840a18f15b8908b84b3d6 2023-03-28 20:54:26,672 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/22a21eb223aa41e882cd90199ba7ea4f 2023-03-28 20:54:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:26,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:27,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036927008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036927010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,110 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/32948b5891a34899a914bfc0ecb3ce48 2023-03-28 20:54:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036927112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,115 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/3340111a5ea840a18f15b8908b84b3d6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6 2023-03-28 20:54:27,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036927114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,120 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6, entries=150, sequenceid=247, filesize=25.3 K 2023-03-28 20:54:27,121 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/22a21eb223aa41e882cd90199ba7ea4f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f 2023-03-28 20:54:27,130 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f, entries=150, sequenceid=247, filesize=11.8 K 2023-03-28 20:54:27,131 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/32948b5891a34899a914bfc0ecb3ce48 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48 2023-03-28 20:54:27,135 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48, entries=150, sequenceid=247, filesize=11.8 K 2023-03-28 20:54:27,138 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 220e271bfe98d8e1f138571bcf50284c in 932ms, sequenceid=247, compaction requested=true 2023-03-28 20:54:27,138 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:27,138 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:27,140 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a716c28[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:27,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:27,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:27,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:27,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:27,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:27,149 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:27,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:27,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:27,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:27,149 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:27,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,155 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,157 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:27,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:27,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:27,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:27,158 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:27,158 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:27,258 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:27,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:27,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:27,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:27,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:27,261 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:27,261 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:27,261 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:27,261 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:27,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,262 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:27,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:27,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:27,266 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:54:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:27,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:27,267 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,294 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032878e7be6a32534025a04c59e6ff6a1d4a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032878e7be6a32534025a04c59e6ff6a1d4a 2023-03-28 20:54:27,308 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/894eb721882c414f950304cce1ff8c54 2023-03-28 20:54:27,320 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0ce38ea6222e4e08863cf9b41365b1e7 2023-03-28 20:54:27,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:27,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:27,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036927343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036927345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49636 deadline: 1680036927447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036927447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49634 deadline: 1680036927650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036927650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,731 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/7fccb0829bef4cf3ac4726565e02eab0 2023-03-28 20:54:27,738 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/894eb721882c414f950304cce1ff8c54 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54 2023-03-28 20:54:27,742 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54, entries=150, sequenceid=261, filesize=25.4 K 2023-03-28 20:54:27,743 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0ce38ea6222e4e08863cf9b41365b1e7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7 2023-03-28 20:54:27,747 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7, entries=150, sequenceid=261, filesize=11.9 K 2023-03-28 20:54:27,748 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/7fccb0829bef4cf3ac4726565e02eab0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0 2023-03-28 20:54:27,752 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0, entries=150, sequenceid=261, filesize=11.9 K 2023-03-28 20:54:27,755 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 220e271bfe98d8e1f138571bcf50284c in 489ms, sequenceid=261, compaction requested=true 2023-03-28 20:54:27,755 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:27,755 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d5f735[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:27,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:27,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,760 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:27,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:27,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:27,764 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:27,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:27,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:27,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:27,765 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:27,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:27,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:27,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:27,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:27,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:27,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:27,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:27,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:27,773 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:27,773 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:27,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:27,873 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:27,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:27,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:27,877 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:27,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:27,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:27,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:27,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,878 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:27,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:27,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:27,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:27,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:27,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:27,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:27,880 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:27,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:27,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:27,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036927966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036927967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:27,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:27,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036927970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036928071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036928072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49594 deadline: 1680036928075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036928274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036928275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036928279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,298 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328d05568d8558a47ed8f5e4c1e86aa4ae8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d05568d8558a47ed8f5e4c1e86aa4ae8 2023-03-28 20:54:28,304 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/d6ea46847d004b099f22334f99cd8376 2023-03-28 20:54:28,314 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/232bdf5f1a004086988c46c110e21c72 2023-03-28 20:54:28,323 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/53cc9eea1b204f63947c5ecf0f325d48 2023-03-28 20:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/d6ea46847d004b099f22334f99cd8376 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376 2023-03-28 20:54:28,333 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376, entries=150, sequenceid=279, filesize=25.4 K 2023-03-28 20:54:28,335 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/232bdf5f1a004086988c46c110e21c72 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72 2023-03-28 20:54:28,340 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72, entries=150, sequenceid=279, filesize=11.9 K 2023-03-28 20:54:28,342 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/53cc9eea1b204f63947c5ecf0f325d48 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48 2023-03-28 20:54:28,346 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48, entries=150, sequenceid=279, filesize=11.9 K 2023-03-28 20:54:28,346 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 220e271bfe98d8e1f138571bcf50284c in 466ms, sequenceid=279, compaction requested=true 2023-03-28 20:54:28,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:28,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:28,349 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326e043c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:28,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:28,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,354 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:28,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:28,356 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:28,358 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:28,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:28,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:28,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:28,358 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:28,360 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,360 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,360 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:28,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:28,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:28,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,366 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:28,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:28,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:28,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:28,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:28,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:28,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,366 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:28,366 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:28,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:28,466 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:28,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:28,470 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:28,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:28,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:28,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:28,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:28,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:28,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:28,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:28,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:28,473 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:28,473 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:28,473 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:54:28,473 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:28,473 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:28,474 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:28,474 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:28,474 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:28,474 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:28,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:28,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:28,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036928601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036928601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036928605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036928706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036928706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036928708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,890 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328663b4d0cb4494336830971d343a5ddf4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328663b4d0cb4494336830971d343a5ddf4 2023-03-28 20:54:28,900 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/d7fe227d54eb41a28d9a43651876268e 2023-03-28 20:54:28,914 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/1a4c6cf6e2b0440392a7fa0c352b26e2 2023-03-28 20:54:28,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036928913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036928914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:28,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036928914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,925 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/6e70c27ad7c04ffa88f578b9a19359db 2023-03-28 20:54:28,929 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/d7fe227d54eb41a28d9a43651876268e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e 2023-03-28 20:54:28,933 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e, entries=150, sequenceid=293, filesize=25.4 K 2023-03-28 20:54:28,933 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/1a4c6cf6e2b0440392a7fa0c352b26e2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2 2023-03-28 20:54:28,937 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2, entries=150, sequenceid=293, filesize=11.9 K 2023-03-28 20:54:28,938 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/6e70c27ad7c04ffa88f578b9a19359db as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db 2023-03-28 20:54:28,941 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db, entries=150, sequenceid=293, filesize=11.9 K 2023-03-28 20:54:28,942 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 220e271bfe98d8e1f138571bcf50284c in 469ms, sequenceid=293, compaction requested=true 2023-03-28 20:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:28,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690226a7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:28,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:28,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:28,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:28,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:28,950 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:28,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:28,951 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:28,951 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:28,951 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:28,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:28,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:28,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:28,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:28,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:28,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:28,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:28,958 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:28,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:28,958 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:28,958 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:28,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:28,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:28,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:29,008 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328c4141c6c6d9a498cb2a0558ea4b8e142 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328c4141c6c6d9a498cb2a0558ea4b8e142 2023-03-28 20:54:29,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036929026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036929029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,058 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,061 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68131783[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,072 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,076 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,076 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,082 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,083 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036929130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036929136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,183 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,186 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,191 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,191 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,191 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,191 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,193 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128529eb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,194 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,194 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,195 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,205 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,205 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,209 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,209 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,209 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,209 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:29,209 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,210 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036929221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036929221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036929221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,310 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,313 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,315 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,317 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,317 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,317 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,317 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,321 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50089063[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,329 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,331 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,332 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,332 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036929332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,336 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,339 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,340 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:29,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,340 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,340 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036929341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,414 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/78b667ff04b145ecb694a30cb9e95beb 2023-03-28 20:54:29,440 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,443 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,448 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2847b3db[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,451 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,451 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,453 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,455 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,455 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,455 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,455 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,455 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,463 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,463 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,463 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,563 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,567 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,569 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,571 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,571 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,571 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,571 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5818165f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,576 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,579 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,580 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,593 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,593 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036929634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036929652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,693 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,697 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,706 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6369b6c6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,708 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,708 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,710 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,713 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,713 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,719 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,720 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,720 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036929723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036929723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:29,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036929724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,820 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,825 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,827 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/051238bfbf934c05a48b77a7ee1130b5 2023-03-28 20:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,830 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:29,830 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:29,831 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f6b940[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:29,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:29,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,835 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:29,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:29,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:29,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/69108d55f8a14b579c1fa2d5b9b74a74 2023-03-28 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:29,841 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:29,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:29,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:29,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:29,841 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:29,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:29,843 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:29,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/78b667ff04b145ecb694a30cb9e95beb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb 2023-03-28 20:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:29,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:29,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:29,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:29,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:29,847 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:29,847 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:29,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:29,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,846 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:29,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:29,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb, entries=200, sequenceid=312, filesize=32.3 K 2023-03-28 20:54:29,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/051238bfbf934c05a48b77a7ee1130b5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5 2023-03-28 20:54:29,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5, entries=150, sequenceid=312, filesize=11.9 K 2023-03-28 20:54:29,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/69108d55f8a14b579c1fa2d5b9b74a74 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74 2023-03-28 20:54:29,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74, entries=150, sequenceid=312, filesize=11.9 K 2023-03-28 20:54:29,860 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 220e271bfe98d8e1f138571bcf50284c in 866ms, sequenceid=312, compaction requested=true 2023-03-28 20:54:29,860 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:29,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:29,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:29,860 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-28 20:54:29,860 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-28 20:54:29,861 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:29,863 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 215066 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-28 20:54:29,863 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:29,863 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97146 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-28 20:54:29,864 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,864 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:29,864 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7efcf89b71c546cda3657d3812ca8576, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=210.0 K 2023-03-28 20:54:29,864 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,864 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/5a7d28da8d1046ccb2433412d1aa4c79, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=94.9 K 2023-03-28 20:54:29,864 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7efcf89b71c546cda3657d3812ca8576, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:29,864 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5a7d28da8d1046ccb2433412d1aa4c79, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:29,864 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 06a1775c0ae34bf89350e8b3096b22a6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680036863078 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 16be89d9b38247d099003b9c59caf9d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680036863078 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 77feea442fc84794a146483bd963ae0a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680036864227 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 69389325d1e54ce5a6550d55c7a7ddf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680036864227 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 3340111a5ea840a18f15b8908b84b3d6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680036864865 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 22a21eb223aa41e882cd90199ba7ea4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680036864865 2023-03-28 20:54:29,865 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 894eb721882c414f950304cce1ff8c54, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680036867005 2023-03-28 20:54:29,866 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0ce38ea6222e4e08863cf9b41365b1e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680036867005 2023-03-28 20:54:29,866 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d6ea46847d004b099f22334f99cd8376, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680036867331 2023-03-28 20:54:29,866 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 232bdf5f1a004086988c46c110e21c72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680036867331 2023-03-28 20:54:29,866 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d7fe227d54eb41a28d9a43651876268e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680036867962 2023-03-28 20:54:29,866 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1a4c6cf6e2b0440392a7fa0c352b26e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680036867962 2023-03-28 20:54:29,867 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 78b667ff04b145ecb694a30cb9e95beb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868595 2023-03-28 20:54:29,867 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 051238bfbf934c05a48b77a7ee1130b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868600 2023-03-28 20:54:29,901 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#174 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:29,910 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#173 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:29,947 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:29,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:29,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:29,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:29,972 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:29,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:29,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:29,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,974 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:29,974 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:29,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:29,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:29,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:29,978 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/44744ed0f3b242e1bc12800f751049a4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/44744ed0f3b242e1bc12800f751049a4 2023-03-28 20:54:29,984 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into 44744ed0f3b242e1bc12800f751049a4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:29,984 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:29,984 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=8, startTime=1680036869860; duration=0sec 2023-03-28 20:54:29,984 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:29,984 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-28 20:54:29,987 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97146 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-28 20:54:29,987 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:29,987 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,988 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/345706e1daa74414a8e772da7e797f1f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=94.9 K 2023-03-28 20:54:29,989 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 345706e1daa74414a8e772da7e797f1f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680036862947 2023-03-28 20:54:29,990 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fcb0751feb834c33be91cdc3710e164c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680036863078 2023-03-28 20:54:29,990 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d35046e9cf114f9ab2194223b925b848, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680036864227 2023-03-28 20:54:29,990 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 32948b5891a34899a914bfc0ecb3ce48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680036864865 2023-03-28 20:54:29,991 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7fccb0829bef4cf3ac4726565e02eab0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680036867005 2023-03-28 20:54:29,991 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 53cc9eea1b204f63947c5ecf0f325d48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680036867331 2023-03-28 20:54:29,992 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6e70c27ad7c04ffa88f578b9a19359db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1680036867962 2023-03-28 20:54:29,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:29,992 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 69108d55f8a14b579c1fa2d5b9b74a74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868600 2023-03-28 20:54:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:29,994 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:29,994 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:29,994 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:29,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:30,014 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#176 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:30,022 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303285e331b1bde044b56803cc2f9df8323de to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e331b1bde044b56803cc2f9df8323de 2023-03-28 20:54:30,052 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8bee8fcc21c64166adb8e699ee5544e5 2023-03-28 20:54:30,071 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/8de5eb3773fc466ba07f996a8a01a86f 2023-03-28 20:54:30,107 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/a0f17b7afbad4c6282bd5ee46f1fcd84 2023-03-28 20:54:30,115 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8bee8fcc21c64166adb8e699ee5544e5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5 2023-03-28 20:54:30,121 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5, entries=150, sequenceid=326, filesize=25.4 K 2023-03-28 20:54:30,122 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/8de5eb3773fc466ba07f996a8a01a86f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f 2023-03-28 20:54:30,127 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f, entries=150, sequenceid=326, filesize=11.9 K 2023-03-28 20:54:30,128 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/a0f17b7afbad4c6282bd5ee46f1fcd84 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84 2023-03-28 20:54:30,159 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84, entries=150, sequenceid=326, filesize=11.9 K 2023-03-28 20:54:30,160 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 220e271bfe98d8e1f138571bcf50284c in 166ms, sequenceid=326, compaction requested=false 2023-03-28 20:54:30,160 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:30,160 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,160 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623a050e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,169 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,169 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,169 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,175 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,175 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,178 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,189 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,190 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:30,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:30,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:30,290 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,296 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,298 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,302 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,302 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,302 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,302 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a884df8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,308 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,308 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,308 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,316 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,317 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036930315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036870300, max timestamp=1680036870317Num uniques 859; 2023-03-28 20:54:30,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:30,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036870300, max timestamp=1680036870317Num uniques 859; 2023-03-28 20:54:30,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:30,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036870300, max timestamp=1680036870317Num uniques 859; 2023-03-28 20:54:30,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:30,320 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,321 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,337 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,337 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,337 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,367 WARN [DataStreamer for file /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328d7b1d229f742473291d735a1fbc28f4c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-28 20:54:30,387 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/709e35367d944d65a3bb1c50776e5a19 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/709e35367d944d65a3bb1c50776e5a19 2023-03-28 20:54:30,391 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into 709e35367d944d65a3bb1c50776e5a19(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:30,391 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:30,391 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=8, startTime=1680036869860; duration=0sec 2023-03-28 20:54:30,391 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:30,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036930409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036930421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,437 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,441 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,452 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/c5c146094613419bbfabfc9669cbfb8e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c5c146094613419bbfabfc9669cbfb8e 2023-03-28 20:54:30,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54791be9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,457 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into c5c146094613419bbfabfc9669cbfb8e(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:30,457 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:30,457 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=8, startTime=1680036869861; duration=0sec 2023-03-28 20:54:30,457 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:30,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,461 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,463 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,463 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,464 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,465 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,465 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,465 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,473 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,473 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,473 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,473 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,473 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:30,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036930514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,573 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,576 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,578 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,581 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,581 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,582 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,582 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1687949f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,586 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,591 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,591 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,593 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,594 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,597 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,597 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036930624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,666 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303280603c52ba2574dea9f51dceae1c43fea to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280603c52ba2574dea9f51dceae1c43fea 2023-03-28 20:54:30,681 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/087476c9a20447d0a0fcdb9d15973856 2023-03-28 20:54:30,697 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,700 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2427e134[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,715 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,715 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,719 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036930718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,723 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,723 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:30,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036930725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036930725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036930737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,823 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,831 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,832 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,832 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,834 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,834 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,834 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,834 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,836 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf88fc5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,844 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,845 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,847 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:30,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,851 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,851 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:30,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036930928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,951 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:30,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:30,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:30,955 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:30,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:30,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:30,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,956 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:30,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:30,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:30,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:30,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:30,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:30,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c279e0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:30,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:30,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:30,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:30,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:30,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:30,970 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:30,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:30,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:30,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:30,970 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:30,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:30,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:30,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:30,982 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,982 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:30,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:30,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:31,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036931024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,082 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:31,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:31,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:31,086 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:31,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:31,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:31,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:31,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:31,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:31,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:31,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:31,089 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,089 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:31,090 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:31,090 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:31,091 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:31,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/424078a82118437c846cb35ea53c3618 2023-03-28 20:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4b260b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:31,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:31,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/393c295ed62141c3abbef07ad4531b68 2023-03-28 20:54:31,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:31,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:31,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:31,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:31,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:31,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:31,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/087476c9a20447d0a0fcdb9d15973856 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856 2023-03-28 20:54:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:31,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856, entries=400, sequenceid=337, filesize=59.8 K 2023-03-28 20:54:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:31,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:31,117 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:31,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:31,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:31,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:31,118 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:31,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/424078a82118437c846cb35ea53c3618 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618 2023-03-28 20:54:31,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:31,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:31,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618, entries=150, sequenceid=337, filesize=11.9 K 2023-03-28 20:54:31,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/393c295ed62141c3abbef07ad4531b68 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68 2023-03-28 20:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,128 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68, entries=150, sequenceid=337, filesize=11.9 K 2023-03-28 20:54:31,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:31,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:31,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:31,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:31,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:31,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:31,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:31,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:31,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:31,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 220e271bfe98d8e1f138571bcf50284c in 891ms, sequenceid=337, compaction requested=true 2023-03-28 20:54:31,132 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:31,132 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:31,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:31,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:31,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:31,132 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:31,132 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:31,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:31,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:31,133 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113926 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:31,133 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:31,133 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:31,133 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,133 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:31,133 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/709e35367d944d65a3bb1c50776e5a19, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=111.3 K 2023-03-28 20:54:31,133 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,134 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/44744ed0f3b242e1bc12800f751049a4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=36.3 K 2023-03-28 20:54:31,134 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 709e35367d944d65a3bb1c50776e5a19, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868600 2023-03-28 20:54:31,134 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 44744ed0f3b242e1bc12800f751049a4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868600 2023-03-28 20:54:31,134 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8bee8fcc21c64166adb8e699ee5544e5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680036869015 2023-03-28 20:54:31,135 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8de5eb3773fc466ba07f996a8a01a86f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680036869015 2023-03-28 20:54:31,135 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 087476c9a20447d0a0fcdb9d15973856, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870161 2023-03-28 20:54:31,135 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 424078a82118437c846cb35ea53c3618, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870230 2023-03-28 20:54:31,142 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:31,148 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#182 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:31,232 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:31,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:31,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:31,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:31,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:31,236 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:31,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:31,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:31,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:31,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:31,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:31,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:31,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:31,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:31,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:31,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:31,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:31,242 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-28 20:54:31,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:31,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:31,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:31,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:31,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036931457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:31,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49650 deadline: 1680036931530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,563 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/38bd645e115a45fa8f9ac53ff580c893 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/38bd645e115a45fa8f9ac53ff580c893 2023-03-28 20:54:31,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036931562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:31,568 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into 38bd645e115a45fa8f9ac53ff580c893(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:31,569 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:31,569 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=13, startTime=1680036871132; duration=0sec 2023-03-28 20:54:31,569 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:31,569 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:31,571 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:31,571 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:31,571 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:31,571 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c5c146094613419bbfabfc9669cbfb8e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=36.3 K 2023-03-28 20:54:31,572 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c5c146094613419bbfabfc9669cbfb8e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680036868600 2023-03-28 20:54:31,573 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a0f17b7afbad4c6282bd5ee46f1fcd84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680036869015 2023-03-28 20:54:31,573 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 393c295ed62141c3abbef07ad4531b68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870230 2023-03-28 20:54:31,583 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:31,586 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/f6ffd5cf966e4a89a1439019b1028522 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/f6ffd5cf966e4a89a1439019b1028522 2023-03-28 20:54:31,596 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into f6ffd5cf966e4a89a1439019b1028522(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:31,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:31,596 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=13, startTime=1680036871132; duration=0sec 2023-03-28 20:54:31,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:31,618 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/1b9c7c87264f484bbfb1a30f128ccce8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/1b9c7c87264f484bbfb1a30f128ccce8 2023-03-28 20:54:31,623 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into 1b9c7c87264f484bbfb1a30f128ccce8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:31,624 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:31,624 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=13, startTime=1680036871132; duration=0sec 2023-03-28 20:54:31,624 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:31,662 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328ee64c534d2504ae29dab4f580d7f9aab to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ee64c534d2504ae29dab4f580d7f9aab 2023-03-28 20:54:31,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:31,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036931768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,068 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8c9e863e4e9943208bd75912caf5ed26 2023-03-28 20:54:32,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49616 deadline: 1680036932070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,081 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/ffc32ad09e894d33906d02f58915b40b 2023-03-28 20:54:32,093 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/921c1dc05eae4cd280f2ac13198d427c 2023-03-28 20:54:32,099 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/8c9e863e4e9943208bd75912caf5ed26 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26 2023-03-28 20:54:32,104 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26, entries=150, sequenceid=375, filesize=25.4 K 2023-03-28 20:54:32,105 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/ffc32ad09e894d33906d02f58915b40b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b 2023-03-28 20:54:32,110 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b, entries=150, sequenceid=375, filesize=11.9 K 2023-03-28 20:54:32,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/921c1dc05eae4cd280f2ac13198d427c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c 2023-03-28 20:54:32,116 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c, entries=150, sequenceid=375, filesize=11.9 K 2023-03-28 20:54:32,117 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 220e271bfe98d8e1f138571bcf50284c in 874ms, sequenceid=375, compaction requested=false 2023-03-28 20:54:32,117 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:32,117 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e4c37[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,125 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,129 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,136 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,136 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:32,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,146 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,146 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:32,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,246 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:32,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:32,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:32,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:32,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:32,249 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:32,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:32,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:32,251 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,251 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,253 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:32,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:32,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:32,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:32,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:32,265 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:54:32,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:32,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:32,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:32,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,310 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303284efef878c638436283cb53dd9079640f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284efef878c638436283cb53dd9079640f 2023-03-28 20:54:32,325 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=389, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/352c4d0cd9cb44cc8d262a63dcb757d2 2023-03-28 20:54:32,358 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/aed97024d781400f9261d24eefeb1a7d 2023-03-28 20:54:32,391 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/68a893390660491ba889b516f6aa2bbf 2023-03-28 20:54:32,396 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/352c4d0cd9cb44cc8d262a63dcb757d2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2 2023-03-28 20:54:32,400 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2, entries=150, sequenceid=389, filesize=25.4 K 2023-03-28 20:54:32,400 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/aed97024d781400f9261d24eefeb1a7d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d 2023-03-28 20:54:32,404 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d, entries=150, sequenceid=389, filesize=11.9 K 2023-03-28 20:54:32,405 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/68a893390660491ba889b516f6aa2bbf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf 2023-03-28 20:54:32,410 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf, entries=150, sequenceid=389, filesize=11.9 K 2023-03-28 20:54:32,411 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 220e271bfe98d8e1f138571bcf50284c in 146ms, sequenceid=389, compaction requested=true 2023-03-28 20:54:32,411 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:32,411 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,421 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69206b7a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,425 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,430 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,430 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,430 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,430 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,430 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,439 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,439 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:32,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,539 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:32,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:32,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:32,542 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:32,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:32,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:32,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,544 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:32,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:32,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:32,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:32,546 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,546 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:32,547 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:32,547 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,549 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7abe02c7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,559 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,560 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,560 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,563 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,563 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,591 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,591 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:32,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:32,612 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:32,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036872647, max timestamp=1680036872661Num uniques 859; 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036872647, max timestamp=1680036872661Num uniques 859; 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036872647, max timestamp=1680036872661Num uniques 859; 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:32,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:32,692 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:32,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:32,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:32,696 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:32,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:32,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:32,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:32,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:32,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:32,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:32,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:32,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036932699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036932701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b608e5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,704 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,704 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,705 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,711 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,711 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,718 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,718 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49636 deadline: 1680036932728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,731 DEBUG [Thread-931] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:32,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49634 deadline: 1680036932738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,740 DEBUG [Thread-927] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:32,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49594 deadline: 1680036932747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,751 DEBUG [Thread-925] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:32,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036932803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:32,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036932803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,818 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:32,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:32,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:32,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:32,822 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:32,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:32,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:32,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:32,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:32,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:32,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:32,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:32,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:32,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:32,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:32,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:32,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,827 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6ae210[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,830 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,837 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,837 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,843 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,843 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,843 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,944 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:32,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:32,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:32,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:32,947 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:32,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:32,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:32,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:32,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:32,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:32,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fb3b9c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:32,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:32,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:32,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,958 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:32,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:32,962 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:32,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:32,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:32,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:32,962 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:32,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:32,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,964 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:32,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:32,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:32,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:32,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:32,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:32,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:32,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:32,978 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:32,978 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:33,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036933006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036933006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,040 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303280ded5da54eb14d6f9a9ebb5a4d901ef0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280ded5da54eb14d6f9a9ebb5a4d901ef0 2023-03-28 20:54:33,048 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/09ac0c920a404a9e9cd065d7ce42e192 2023-03-28 20:54:33,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/623f99db510e4e33aa7b2cb725e34be7 2023-03-28 20:54:33,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/8ab1fb7f7c6d4e548e7c4f8edcee541e 2023-03-28 20:54:33,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/09ac0c920a404a9e9cd065d7ce42e192 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192 2023-03-28 20:54:33,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192, entries=350, sequenceid=400, filesize=52.9 K 2023-03-28 20:54:33,078 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:33,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:33,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/623f99db510e4e33aa7b2cb725e34be7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7 2023-03-28 20:54:33,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:33,082 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:33,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:33,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:33,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:33,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7, entries=150, sequenceid=400, filesize=11.9 K 2023-03-28 20:54:33,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:33,084 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:33,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:33,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/8ab1fb7f7c6d4e548e7c4f8edcee541e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e 2023-03-28 20:54:33,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:33,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:33,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:33,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:33,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:33,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:33,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:33,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e, entries=150, sequenceid=400, filesize=11.9 K 2023-03-28 20:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad9ad7e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:33,092 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:33,092 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 220e271bfe98d8e1f138571bcf50284c in 481ms, sequenceid=400, compaction requested=true 2023-03-28 20:54:33,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:33,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:33,093 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:33,093 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:33,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:33,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:33,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,094 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 133034 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:33,094 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:33,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:33,095 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:33,095 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:33,095 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,095 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/38bd645e115a45fa8f9ac53ff580c893, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=48.3 K 2023-03-28 20:54:33,095 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,095 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/f6ffd5cf966e4a89a1439019b1028522, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=129.9 K 2023-03-28 20:54:33,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,096 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f6ffd5cf966e4a89a1439019b1028522, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870230 2023-03-28 20:54:33,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:33,096 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 38bd645e115a45fa8f9ac53ff580c893, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870230 2023-03-28 20:54:33,097 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8c9e863e4e9943208bd75912caf5ed26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036870364 2023-03-28 20:54:33,097 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ffc32ad09e894d33906d02f58915b40b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036870364 2023-03-28 20:54:33,097 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 352c4d0cd9cb44cc8d262a63dcb757d2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680036871451 2023-03-28 20:54:33,097 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting aed97024d781400f9261d24eefeb1a7d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680036871451 2023-03-28 20:54:33,098 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 09ac0c920a404a9e9cd065d7ce42e192, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872559 2023-03-28 20:54:33,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:33,098 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 623f99db510e4e33aa7b2cb725e34be7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872608 2023-03-28 20:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:33,101 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:33,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:33,102 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:33,103 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:33,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:33,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:33,106 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:33,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:33,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:33,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:33,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:33,109 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:33,109 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:33,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:33,138 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#195 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:33,181 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#194 average throughput is 0.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:33,187 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0cc3ca82d7924aebb30a6cc8f1e603ad as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0cc3ca82d7924aebb30a6cc8f1e603ad 2023-03-28 20:54:33,192 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into 0cc3ca82d7924aebb30a6cc8f1e603ad(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:33,192 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:33,192 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=12, startTime=1680036873093; duration=0sec 2023-03-28 20:54:33,192 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:33,192 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:33,195 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:33,195 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:33,195 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,195 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/1b9c7c87264f484bbfb1a30f128ccce8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=48.3 K 2023-03-28 20:54:33,195 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1b9c7c87264f484bbfb1a30f128ccce8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680036870230 2023-03-28 20:54:33,196 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 921c1dc05eae4cd280f2ac13198d427c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036870364 2023-03-28 20:54:33,196 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 68a893390660491ba889b516f6aa2bbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680036871451 2023-03-28 20:54:33,196 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8ab1fb7f7c6d4e548e7c4f8edcee541e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872608 2023-03-28 20:54:33,209 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:33,211 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#196 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:33,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:33,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:33,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:33,220 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:33,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:33,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:33,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:33,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,221 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,223 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:33,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:33,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:33,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:33,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:33,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:33,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:33,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:33,228 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:33,228 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:33,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:33,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:33,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036933318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036933321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036933422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036933424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,623 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/398d21540f8b448ea379faef7289c0e5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/398d21540f8b448ea379faef7289c0e5 2023-03-28 20:54:33,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036933625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036933627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,635 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/2fe62d06b4dd4d64baa9f99abfb443d8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/2fe62d06b4dd4d64baa9f99abfb443d8 2023-03-28 20:54:33,640 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into 398d21540f8b448ea379faef7289c0e5(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:33,640 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:33,640 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=12, startTime=1680036873093; duration=0sec 2023-03-28 20:54:33,640 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:33,645 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into 2fe62d06b4dd4d64baa9f99abfb443d8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:33,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:33,645 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=12, startTime=1680036873093; duration=0sec 2023-03-28 20:54:33,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:33,651 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032875c6a6e963094451b14a56b8fac314ad to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875c6a6e963094451b14a56b8fac314ad 2023-03-28 20:54:33,666 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/fc0589922ab042b4b89a625038782227 2023-03-28 20:54:33,682 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/136a449a85af415e89236113aeb1d9cf 2023-03-28 20:54:33,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036933931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:33,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:33,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036933931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,098 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/ea25993aedc74739baa39cc5743ff959 2023-03-28 20:54:34,102 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/fc0589922ab042b4b89a625038782227 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227 2023-03-28 20:54:34,108 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227, entries=150, sequenceid=436, filesize=25.4 K 2023-03-28 20:54:34,110 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/136a449a85af415e89236113aeb1d9cf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf 2023-03-28 20:54:34,121 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:54:34,122 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/ea25993aedc74739baa39cc5743ff959 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959 2023-03-28 20:54:34,126 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:54:34,127 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 220e271bfe98d8e1f138571bcf50284c in 899ms, sequenceid=436, compaction requested=false 2023-03-28 20:54:34,127 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:34,127 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:34,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139169e0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:34,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:34,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:34,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:34,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:34,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:34,138 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:34,141 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:34,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:34,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:34,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:34,142 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:34,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,145 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:34,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,151 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:34,151 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:34,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:34,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,252 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:34,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:34,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:34,256 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:34,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:34,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:34,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,259 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,259 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:34,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:34,261 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:34,261 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:34,261 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:34,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:34,262 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:34,262 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:34,278 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328371e7f4cf1904f48a11e0c2fb930c8fd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328371e7f4cf1904f48a11e0c2fb930c8fd 2023-03-28 20:54:34,295 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=448, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/7b946be0abb843cc91a139ea04114ef1 2023-03-28 20:54:34,309 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/a1284e7d4f6842fb9ea5332588d76f0e 2023-03-28 20:54:34,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:34,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036874447, max timestamp=1680036874461Num uniques 859; 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036874447, max timestamp=1680036874461Num uniques 859; 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036874447, max timestamp=1680036874461Num uniques 859; 2023-03-28 20:54:34,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:54:34,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036934492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036934493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036934596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036934597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,719 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/99350c2b77e0432f9cfd4b9165b8507d 2023-03-28 20:54:34,723 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/7b946be0abb843cc91a139ea04114ef1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1 2023-03-28 20:54:34,727 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1, entries=150, sequenceid=448, filesize=25.4 K 2023-03-28 20:54:34,728 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/a1284e7d4f6842fb9ea5332588d76f0e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e 2023-03-28 20:54:34,731 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e, entries=150, sequenceid=448, filesize=11.9 K 2023-03-28 20:54:34,731 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/99350c2b77e0432f9cfd4b9165b8507d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d 2023-03-28 20:54:34,735 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d, entries=150, sequenceid=448, filesize=11.9 K 2023-03-28 20:54:34,736 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 220e271bfe98d8e1f138571bcf50284c in 475ms, sequenceid=448, compaction requested=true 2023-03-28 20:54:34,736 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:34,736 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,736 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:34,736 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:34,736 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,736 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:34,736 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:34,737 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7673e0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:34,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:34,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,741 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:34,742 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:34,742 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:34,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:34,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:34,746 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:34,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:34,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:34,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:34,746 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:34,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,749 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:34,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:34,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,758 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:34,758 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:34,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:34,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:34,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:34,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:34,815 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303287a0bfd309e2244e0bf9d22fe9c83d7ec to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287a0bfd309e2244e0bf9d22fe9c83d7ec 2023-03-28 20:54:34,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036934831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036934831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,858 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:34,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:34,861 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:34,861 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,861 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:34,861 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:34,861 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,861 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:34,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:34,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:34,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:34,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:34,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:34,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:34,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:34,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a89e69[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:34,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:34,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,869 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:34,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:34,870 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,870 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:34,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:34,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:34,874 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:34,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:34,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:34,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:34,875 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:34,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:34,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:34,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:34,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,880 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:34,880 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036934937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036934937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,980 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:34,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:34,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:34,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:34,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:34,983 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:34,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:34,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:34,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,986 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:34,987 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,987 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:34,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:34,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:34,988 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,988 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:34,988 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:34,989 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,990 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107d7fbf[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:34,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:34,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:34,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:34,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:34,995 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:34,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:34,998 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:34,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:34,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:34,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:34,998 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,001 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,001 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,001 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,001 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,002 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,004 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,005 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,005 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,105 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,108 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83bca39[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,117 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,121 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,121 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,123 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,125 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,129 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,130 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,130 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036935141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036935141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,226 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=485, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/11a9c947901a469c8abdc45ff5a277d6 2023-03-28 20:54:35,230 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,234 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,239 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,240 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,240 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,240 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,240 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ef41ad[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,248 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,252 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,252 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,252 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,252 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,252 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,254 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,254 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,254 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,257 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,258 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,258 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,358 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,361 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,367 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@687daaf[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,372 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,372 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,372 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,377 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,377 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,378 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,379 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,381 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,382 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,382 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036935442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036935442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,482 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,486 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,487 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,490 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,490 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,490 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,490 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,492 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfccd47[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,493 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,493 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,498 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,499 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,501 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,505 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,505 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,506 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,506 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,605 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,618 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,620 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc0aac3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/f49966c253414ef4a09094983a1119fc 2023-03-28 20:54:35,638 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,641 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,642 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,642 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,642 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,644 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,644 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,644 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,645 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,653 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,653 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,753 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,757 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,766 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ed643f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,767 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,767 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,769 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,769 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,769 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,770 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,773 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,773 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,781 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,781 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,781 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,881 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:35,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:35,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:35,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:35,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:35,884 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:35,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:35,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:35,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:35,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:35,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:35,888 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,888 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:35,888 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:35,888 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:35,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0788af[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:35,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:35,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:35,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:35,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:35,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:35,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:35,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:35,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:35,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:35,898 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:35,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:35,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:35,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:35,898 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:35,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:35,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:35,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:35,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:35,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:35,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:35,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:35,905 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:35,905 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:35,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:35,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:35,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:35,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49616 deadline: 1680036935946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:35,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:35,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49650 deadline: 1680036935949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,005 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:36,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:36,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:36,008 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:36,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:36,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:36,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:36,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,009 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:36,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:36,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:36,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:36,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:36,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:36,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:36,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:36,012 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,012 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:36,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:36,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:36,014 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:36,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470ae09d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:36,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:36,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,017 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:36,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:36,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:36,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:36,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:36,025 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:36,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:36,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:36,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:36,025 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:36,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:36,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:36,027 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:36,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:36,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:36,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:36,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:36,030 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:36,031 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:36,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:36,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/32a0b887d72e4c95a6d69feb88f1ba09 2023-03-28 20:54:36,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/11a9c947901a469c8abdc45ff5a277d6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6 2023-03-28 20:54:36,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6, entries=150, sequenceid=485, filesize=25.4 K 2023-03-28 20:54:36,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/f49966c253414ef4a09094983a1119fc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc 2023-03-28 20:54:36,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc, entries=150, sequenceid=485, filesize=11.9 K 2023-03-28 20:54:36,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/32a0b887d72e4c95a6d69feb88f1ba09 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09 2023-03-28 20:54:36,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09, entries=150, sequenceid=485, filesize=11.9 K 2023-03-28 20:54:36,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=40.25 KB/41220 for 220e271bfe98d8e1f138571bcf50284c in 1284ms, sequenceid=485, compaction requested=true 2023-03-28 20:54:36,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:36,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:36,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:36,087 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:36,087 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:36,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:36,088 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:36,088 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:36,088 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/B is initiating minor compaction (all files) 2023-03-28 20:54:36,088 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/A is initiating minor compaction (all files) 2023-03-28 20:54:36,088 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/B in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,088 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/A in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,088 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0cc3ca82d7924aebb30a6cc8f1e603ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=48.4 K 2023-03-28 20:54:36,088 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/398d21540f8b448ea379faef7289c0e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=102.5 K 2023-03-28 20:54:36,089 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0cc3ca82d7924aebb30a6cc8f1e603ad, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872608 2023-03-28 20:54:36,089 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 398d21540f8b448ea379faef7289c0e5, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872608 2023-03-28 20:54:36,089 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 136a449a85af415e89236113aeb1d9cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036872687 2023-03-28 20:54:36,089 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fc0589922ab042b4b89a625038782227, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036872687 2023-03-28 20:54:36,090 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a1284e7d4f6842fb9ea5332588d76f0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680036873314 2023-03-28 20:54:36,090 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7b946be0abb843cc91a139ea04114ef1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680036873314 2023-03-28 20:54:36,090 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f49966c253414ef4a09094983a1119fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680036874492 2023-03-28 20:54:36,090 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 11a9c947901a469c8abdc45ff5a277d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680036874492 2023-03-28 20:54:36,126 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:36,131 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:36,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:36,148 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#A#compaction#206 average throughput is 0.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:36,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0e18321c4fee44c2903b9a40246b14e7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0e18321c4fee44c2903b9a40246b14e7 2023-03-28 20:54:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:36,163 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:36,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:36,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:36,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:36,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,165 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/B of 220e271bfe98d8e1f138571bcf50284c into 0e18321c4fee44c2903b9a40246b14e7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:36,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:36,165 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/B, priority=12, startTime=1680036876087; duration=0sec 2023-03-28 20:54:36,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:36,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:36,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:36,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:36,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:36,167 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:36,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:36,167 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 220e271bfe98d8e1f138571bcf50284c/C is initiating minor compaction (all files) 2023-03-28 20:54:36,168 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 220e271bfe98d8e1f138571bcf50284c/C in TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,168 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/2fe62d06b4dd4d64baa9f99abfb443d8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp, totalSize=48.4 K 2023-03-28 20:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:36,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:36,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2fe62d06b4dd4d64baa9f99abfb443d8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680036872608 2023-03-28 20:54:36,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ea25993aedc74739baa39cc5743ff959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036872687 2023-03-28 20:54:36,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 99350c2b77e0432f9cfd4b9165b8507d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680036873314 2023-03-28 20:54:36,170 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 32a0b887d72e4c95a6d69feb88f1ba09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680036874492 2023-03-28 20:54:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:36,172 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:36,172 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. started... 2023-03-28 20:54:36,173 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:36,173 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:36,181 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 220e271bfe98d8e1f138571bcf50284c#C#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:36,210 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/f28383a974f64a11a3a7a7119fec06f4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f28383a974f64a11a3a7a7119fec06f4 2023-03-28 20:54:36,221 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/C of 220e271bfe98d8e1f138571bcf50284c into f28383a974f64a11a3a7a7119fec06f4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:36,221 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:36,221 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/C, priority=12, startTime=1680036876087; duration=0sec 2023-03-28 20:54:36,221 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:36,556 DEBUG [Thread-934] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:56827 2023-03-28 20:54:36,556 DEBUG [Thread-934] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,557 DEBUG [Thread-938] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1345b761 to 127.0.0.1:56827 2023-03-28 20:54:36,557 DEBUG [Thread-938] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,558 DEBUG [Thread-936] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63d375f8 to 127.0.0.1:56827 2023-03-28 20:54:36,558 DEBUG [Thread-936] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,559 DEBUG [Thread-940] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e0b0a9f to 127.0.0.1:56827 2023-03-28 20:54:36,559 DEBUG [Thread-940] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/c54ff03af6b049c68b3b907d465a8afd as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c54ff03af6b049c68b3b907d465a8afd 2023-03-28 20:54:36,600 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303285993c387840941e9a843932da6117027 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285993c387840941e9a843932da6117027 2023-03-28 20:54:36,601 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 220e271bfe98d8e1f138571bcf50284c/A of 220e271bfe98d8e1f138571bcf50284c into c54ff03af6b049c68b3b907d465a8afd(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:36,601 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:36,601 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c., storeName=220e271bfe98d8e1f138571bcf50284c/A, priority=12, startTime=1680036876086; duration=0sec 2023-03-28 20:54:36,601 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:36,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. as already flushing 2023-03-28 20:54:36,744 DEBUG [Thread-931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76aef981 to 127.0.0.1:56827 2023-03-28 20:54:36,744 DEBUG [Thread-931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,781 DEBUG [Thread-927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28fcbe97 to 127.0.0.1:56827 2023-03-28 20:54:36,781 DEBUG [Thread-927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,783 DEBUG [Thread-925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46078b1e to 127.0.0.1:56827 2023-03-28 20:54:36,783 DEBUG [Thread-925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,955 DEBUG [Thread-923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x542e33ab to 127.0.0.1:56827 2023-03-28 20:54:36,955 DEBUG [Thread-929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:56827 2023-03-28 20:54:36,955 DEBUG [Thread-923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:36,955 DEBUG [Thread-929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:37,005 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=495, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/665ec74a37c444b3a56d5e4ef10465bd 2023-03-28 20:54:37,414 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/d735916559c7439f80ed67b82a1e9680 2023-03-28 20:54:37,422 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/e9de594ae3a346c4888c0ad55f6d69ef 2023-03-28 20:54:37,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/665ec74a37c444b3a56d5e4ef10465bd as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/665ec74a37c444b3a56d5e4ef10465bd 2023-03-28 20:54:37,428 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/665ec74a37c444b3a56d5e4ef10465bd, entries=150, sequenceid=495, filesize=25.4 K 2023-03-28 20:54:37,429 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/d735916559c7439f80ed67b82a1e9680 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d735916559c7439f80ed67b82a1e9680 2023-03-28 20:54:37,432 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d735916559c7439f80ed67b82a1e9680, entries=150, sequenceid=495, filesize=11.9 K 2023-03-28 20:54:37,433 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/e9de594ae3a346c4888c0ad55f6d69ef as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9de594ae3a346c4888c0ad55f6d69ef 2023-03-28 20:54:37,436 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9de594ae3a346c4888c0ad55f6d69ef, entries=150, sequenceid=495, filesize=11.9 K 2023-03-28 20:54:37,436 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=33.54 KB/34350 for 220e271bfe98d8e1f138571bcf50284c in 1264ms, sequenceid=495, compaction requested=false 2023-03-28 20:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:37,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b2dc0d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:37,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:37,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:37,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:37,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:37,444 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:37,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:37,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:37,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:37,444 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:37,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:37,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,448 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:37,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:37,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:37,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:37,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:37,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:37,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:37,450 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:37,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:37,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:37,451 DEBUG [Thread-933] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:37,451 DEBUG [Thread-933] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:37,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:37,551 DEBUG [Thread-933] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 4613 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 4775 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1993 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5979 rows 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1973 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5919 rows 2023-03-28 20:54:37,552 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:54:37,552 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58d22e5a to 127.0.0.1:56827 2023-03-28 20:54:37,553 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:54:37,554 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:54:37,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:54:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:37,559 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036877559"}]},"ts":"1680036877559"} 2023-03-28 20:54:37,560 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:54:37,563 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:54:37,564 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, UNASSIGN}] 2023-03-28 20:54:37,565 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, UNASSIGN 2023-03-28 20:54:37,566 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:37,566 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036877566"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036877566"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036877566"}]},"ts":"1680036877566"} 2023-03-28 20:54:37,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:37,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:37,664 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-28 20:54:37,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 220e271bfe98d8e1f138571bcf50284c, disabling compactions & flushes 2023-03-28 20:54:37,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. after waiting 0 ms 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:37,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 220e271bfe98d8e1f138571bcf50284c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=A 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=B 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 220e271bfe98d8e1f138571bcf50284c, store=C 2023-03-28 20:54:37,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:37,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:38,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328dbc8a1327e4841db9e36ed5fd077b4f7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328dbc8a1327e4841db9e36ed5fd077b4f7 2023-03-28 20:54:38,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:38,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=505, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/df078dcfd8c04b14a07ccedd6ad2a0c2 2023-03-28 20:54:38,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:38,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff 2023-03-28 20:54:39,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/3404fe98484a42d2975fdf0a6624c223 2023-03-28 20:54:39,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/A/df078dcfd8c04b14a07ccedd6ad2a0c2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/df078dcfd8c04b14a07ccedd6ad2a0c2 2023-03-28 20:54:39,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/df078dcfd8c04b14a07ccedd6ad2a0c2, entries=150, sequenceid=505, filesize=25.4 K 2023-03-28 20:54:39,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff 2023-03-28 20:54:39,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff, entries=150, sequenceid=505, filesize=11.9 K 2023-03-28 20:54:39,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/.tmp/C/3404fe98484a42d2975fdf0a6624c223 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/3404fe98484a42d2975fdf0a6624c223 2023-03-28 20:54:39,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/3404fe98484a42d2975fdf0a6624c223, entries=150, sequenceid=505, filesize=11.9 K 2023-03-28 20:54:39,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 220e271bfe98d8e1f138571bcf50284c in 1650ms, sequenceid=505, compaction requested=true 2023-03-28 20:54:39,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7ce45558e2aa4da598dd7bf4c555446d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7efcf89b71c546cda3657d3812ca8576, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/709e35367d944d65a3bb1c50776e5a19, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/f6ffd5cf966e4a89a1439019b1028522, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/398d21540f8b448ea379faef7289c0e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6] to archive 2023-03-28 20:54:39,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:39,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c5ac10000a794e548705ba2582246df9 2023-03-28 20:54:39,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/bfe1a7bac04f48afab763c3f0b64c5f3 2023-03-28 20:54:39,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/0345a935d2744891aa46ec45811bb7ad 2023-03-28 20:54:39,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/96083869ea02429fa1c4d50421de35a9 2023-03-28 20:54:39,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8e4e6ddec3ad4fcdbe4b70006c4344bf 2023-03-28 20:54:39,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/ea0343d5b0ef4e02bee27190c6260026 2023-03-28 20:54:39,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7ce45558e2aa4da598dd7bf4c555446d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7ce45558e2aa4da598dd7bf4c555446d 2023-03-28 20:54:39,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3dfbcb6f7b9b4003a922c1d6a75b74d4 2023-03-28 20:54:39,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/26d19e186ac3458993c1ec47497ba52b 2023-03-28 20:54:39,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7efcf89b71c546cda3657d3812ca8576 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7efcf89b71c546cda3657d3812ca8576 2023-03-28 20:54:39,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/06a1775c0ae34bf89350e8b3096b22a6 2023-03-28 20:54:39,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/77feea442fc84794a146483bd963ae0a 2023-03-28 20:54:39,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/3340111a5ea840a18f15b8908b84b3d6 2023-03-28 20:54:39,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/894eb721882c414f950304cce1ff8c54 2023-03-28 20:54:39,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d6ea46847d004b099f22334f99cd8376 2023-03-28 20:54:39,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/d7fe227d54eb41a28d9a43651876268e 2023-03-28 20:54:39,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/78b667ff04b145ecb694a30cb9e95beb 2023-03-28 20:54:39,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/709e35367d944d65a3bb1c50776e5a19 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/709e35367d944d65a3bb1c50776e5a19 2023-03-28 20:54:39,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8bee8fcc21c64166adb8e699ee5544e5 2023-03-28 20:54:39,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/087476c9a20447d0a0fcdb9d15973856 2023-03-28 20:54:39,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/f6ffd5cf966e4a89a1439019b1028522 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/f6ffd5cf966e4a89a1439019b1028522 2023-03-28 20:54:39,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/8c9e863e4e9943208bd75912caf5ed26 2023-03-28 20:54:39,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/352c4d0cd9cb44cc8d262a63dcb757d2 2023-03-28 20:54:39,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/09ac0c920a404a9e9cd065d7ce42e192 2023-03-28 20:54:39,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/398d21540f8b448ea379faef7289c0e5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/398d21540f8b448ea379faef7289c0e5 2023-03-28 20:54:39,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/fc0589922ab042b4b89a625038782227 2023-03-28 20:54:39,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/7b946be0abb843cc91a139ea04114ef1 2023-03-28 20:54:39,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/11a9c947901a469c8abdc45ff5a277d6 2023-03-28 20:54:39,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/b8def24c8b3c4d94b8fcefcd96f6ef8d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/5a7d28da8d1046ccb2433412d1aa4c79, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/44744ed0f3b242e1bc12800f751049a4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/38bd645e115a45fa8f9ac53ff580c893, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0cc3ca82d7924aebb30a6cc8f1e603ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc] to archive 2023-03-28 20:54:39,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:39,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8ed4bbae18b84c81bb28ccdcd06c2a31 2023-03-28 20:54:39,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/671db35c06204f4da6011303e5c0acfb 2023-03-28 20:54:39,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f4a45a9114cb4794aa7cf613af30dc29 2023-03-28 20:54:39,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/014e001552bf420f93035729e708cec1 2023-03-28 20:54:39,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d449d7ef15844176a0b3a42bb735c794 2023-03-28 20:54:39,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/b8def24c8b3c4d94b8fcefcd96f6ef8d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/b8def24c8b3c4d94b8fcefcd96f6ef8d 2023-03-28 20:54:39,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/90c1415c669043adbe23bdb246ed805d 2023-03-28 20:54:39,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/4b4a7598589c4369980043d8bf037450 2023-03-28 20:54:39,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/5a7d28da8d1046ccb2433412d1aa4c79 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/5a7d28da8d1046ccb2433412d1aa4c79 2023-03-28 20:54:39,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ce8e20f7328745afac7a6d950d18bdf1 2023-03-28 20:54:39,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/16be89d9b38247d099003b9c59caf9d0 2023-03-28 20:54:39,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/69389325d1e54ce5a6550d55c7a7ddf9 2023-03-28 20:54:39,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/22a21eb223aa41e882cd90199ba7ea4f 2023-03-28 20:54:39,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0ce38ea6222e4e08863cf9b41365b1e7 2023-03-28 20:54:39,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/232bdf5f1a004086988c46c110e21c72 2023-03-28 20:54:39,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/1a4c6cf6e2b0440392a7fa0c352b26e2 2023-03-28 20:54:39,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/44744ed0f3b242e1bc12800f751049a4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/44744ed0f3b242e1bc12800f751049a4 2023-03-28 20:54:39,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/051238bfbf934c05a48b77a7ee1130b5 2023-03-28 20:54:39,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/8de5eb3773fc466ba07f996a8a01a86f 2023-03-28 20:54:39,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/38bd645e115a45fa8f9ac53ff580c893 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/38bd645e115a45fa8f9ac53ff580c893 2023-03-28 20:54:39,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/424078a82118437c846cb35ea53c3618 2023-03-28 20:54:39,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/ffc32ad09e894d33906d02f58915b40b 2023-03-28 20:54:39,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/aed97024d781400f9261d24eefeb1a7d 2023-03-28 20:54:39,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0cc3ca82d7924aebb30a6cc8f1e603ad to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0cc3ca82d7924aebb30a6cc8f1e603ad 2023-03-28 20:54:39,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/623f99db510e4e33aa7b2cb725e34be7 2023-03-28 20:54:39,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/136a449a85af415e89236113aeb1d9cf 2023-03-28 20:54:39,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/a1284e7d4f6842fb9ea5332588d76f0e 2023-03-28 20:54:39,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/f49966c253414ef4a09094983a1119fc 2023-03-28 20:54:39,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/dab8ddc60b7c4a83a72765906f1634ac, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/345706e1daa74414a8e772da7e797f1f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c5c146094613419bbfabfc9669cbfb8e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/1b9c7c87264f484bbfb1a30f128ccce8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/2fe62d06b4dd4d64baa9f99abfb443d8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09] to archive 2023-03-28 20:54:39,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:54:39,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d898cdbc4a37404db92e920831cf4936 2023-03-28 20:54:39,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9d6cdb103674270a5dd35b25e809b90 2023-03-28 20:54:39,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/94384d72d3c04334b77673a9982dcd7c 2023-03-28 20:54:39,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/cdc248d03ecd45238b7a9458062aa7f6 2023-03-28 20:54:39,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f7d86a007b8843d2a7f2afd092be9c39 2023-03-28 20:54:39,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/dab8ddc60b7c4a83a72765906f1634ac to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/dab8ddc60b7c4a83a72765906f1634ac 2023-03-28 20:54:39,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9fadc091d6774bcb82fe3b9a976bf7eb 2023-03-28 20:54:39,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c768e5fffd3c42659a12ad5a97ec893b 2023-03-28 20:54:39,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/345706e1daa74414a8e772da7e797f1f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/345706e1daa74414a8e772da7e797f1f 2023-03-28 20:54:39,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/9967093c989343ecaae1299f83a8b30c 2023-03-28 20:54:39,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/fcb0751feb834c33be91cdc3710e164c 2023-03-28 20:54:39,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/d35046e9cf114f9ab2194223b925b848 2023-03-28 20:54:39,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32948b5891a34899a914bfc0ecb3ce48 2023-03-28 20:54:39,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/7fccb0829bef4cf3ac4726565e02eab0 2023-03-28 20:54:39,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/53cc9eea1b204f63947c5ecf0f325d48 2023-03-28 20:54:39,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/6e70c27ad7c04ffa88f578b9a19359db 2023-03-28 20:54:39,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c5c146094613419bbfabfc9669cbfb8e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/c5c146094613419bbfabfc9669cbfb8e 2023-03-28 20:54:39,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/69108d55f8a14b579c1fa2d5b9b74a74 2023-03-28 20:54:39,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/a0f17b7afbad4c6282bd5ee46f1fcd84 2023-03-28 20:54:39,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/1b9c7c87264f484bbfb1a30f128ccce8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/1b9c7c87264f484bbfb1a30f128ccce8 2023-03-28 20:54:39,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/393c295ed62141c3abbef07ad4531b68 2023-03-28 20:54:39,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/921c1dc05eae4cd280f2ac13198d427c 2023-03-28 20:54:39,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/68a893390660491ba889b516f6aa2bbf 2023-03-28 20:54:39,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/2fe62d06b4dd4d64baa9f99abfb443d8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/2fe62d06b4dd4d64baa9f99abfb443d8 2023-03-28 20:54:39,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/8ab1fb7f7c6d4e548e7c4f8edcee541e 2023-03-28 20:54:39,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/ea25993aedc74739baa39cc5743ff959 2023-03-28 20:54:39,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/99350c2b77e0432f9cfd4b9165b8507d 2023-03-28 20:54:39,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/32a0b887d72e4c95a6d69feb88f1ba09 2023-03-28 20:54:39,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits/508.seqid, newMaxSeqId=508, maxSeqId=4 2023-03-28 20:54:39,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c. 2023-03-28 20:54:39,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 220e271bfe98d8e1f138571bcf50284c: 2023-03-28 20:54:39,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:39,485 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=220e271bfe98d8e1f138571bcf50284c, regionState=CLOSED 2023-03-28 20:54:39,485 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036879485"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036879485"}]},"ts":"1680036879485"} 2023-03-28 20:54:39,487 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-28 20:54:39,487 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 220e271bfe98d8e1f138571bcf50284c, server=jenkins-hbase4.apache.org,42693,1680036815956 in 1.9190 sec 2023-03-28 20:54:39,488 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-28 20:54:39,488 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=220e271bfe98d8e1f138571bcf50284c, UNASSIGN in 1.9230 sec 2023-03-28 20:54:39,489 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036879489"}]},"ts":"1680036879489"} 2023-03-28 20:54:39,490 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:54:39,491 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:54:39,492 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9360 sec 2023-03-28 20:54:39,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-28 20:54:39,663 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-28 20:54:39,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:54:39,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,666 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-28 20:54:39,667 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,671 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:39,674 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits] 2023-03-28 20:54:39,676 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/665ec74a37c444b3a56d5e4ef10465bd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/665ec74a37c444b3a56d5e4ef10465bd 2023-03-28 20:54:39,678 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c54ff03af6b049c68b3b907d465a8afd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/c54ff03af6b049c68b3b907d465a8afd 2023-03-28 20:54:39,679 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/df078dcfd8c04b14a07ccedd6ad2a0c2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/A/df078dcfd8c04b14a07ccedd6ad2a0c2 2023-03-28 20:54:39,681 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0b48a3b69cfd48d7b675ae3cf3e0a4ff 2023-03-28 20:54:39,682 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0e18321c4fee44c2903b9a40246b14e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/0e18321c4fee44c2903b9a40246b14e7 2023-03-28 20:54:39,683 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d735916559c7439f80ed67b82a1e9680 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/B/d735916559c7439f80ed67b82a1e9680 2023-03-28 20:54:39,685 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/3404fe98484a42d2975fdf0a6624c223 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/3404fe98484a42d2975fdf0a6624c223 2023-03-28 20:54:39,686 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9de594ae3a346c4888c0ad55f6d69ef to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/e9de594ae3a346c4888c0ad55f6d69ef 2023-03-28 20:54:39,687 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f28383a974f64a11a3a7a7119fec06f4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/C/f28383a974f64a11a3a7a7119fec06f4 2023-03-28 20:54:39,690 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits/508.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c/recovered.edits/508.seqid 2023-03-28 20:54:39,690 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/220e271bfe98d8e1f138571bcf50284c 2023-03-28 20:54:39,690 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:54:39,691 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:54:39,691 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-28 20:54:39,695 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032802219e0165494467af1974aab7e00886 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032802219e0165494467af1974aab7e00886 2023-03-28 20:54:39,696 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280603c52ba2574dea9f51dceae1c43fea to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280603c52ba2574dea9f51dceae1c43fea 2023-03-28 20:54:39,697 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280ded5da54eb14d6f9a9ebb5a4d901ef0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280ded5da54eb14d6f9a9ebb5a4d901ef0 2023-03-28 20:54:39,698 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282a509616bea14a6f9cda4dbe457eb0d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282a509616bea14a6f9cda4dbe457eb0d0 2023-03-28 20:54:39,699 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328371e7f4cf1904f48a11e0c2fb930c8fd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328371e7f4cf1904f48a11e0c2fb930c8fd 2023-03-28 20:54:39,700 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328444be8ae14414aaa9c9d18db0ebff599 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328444be8ae14414aaa9c9d18db0ebff599 2023-03-28 20:54:39,701 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032844eae8b4a84d4da9a98207180828f7d5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032844eae8b4a84d4da9a98207180828f7d5 2023-03-28 20:54:39,702 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328458578387d944afa910aff6545c1df1c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328458578387d944afa910aff6545c1df1c 2023-03-28 20:54:39,703 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284a15ef66fa654ee9af8dc3415071cd4e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284a15ef66fa654ee9af8dc3415071cd4e 2023-03-28 20:54:39,704 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284efef878c638436283cb53dd9079640f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284efef878c638436283cb53dd9079640f 2023-03-28 20:54:39,705 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285993c387840941e9a843932da6117027 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285993c387840941e9a843932da6117027 2023-03-28 20:54:39,706 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e331b1bde044b56803cc2f9df8323de to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e331b1bde044b56803cc2f9df8323de 2023-03-28 20:54:39,707 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328663b4d0cb4494336830971d343a5ddf4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328663b4d0cb4494336830971d343a5ddf4 2023-03-28 20:54:39,708 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875c6a6e963094451b14a56b8fac314ad to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875c6a6e963094451b14a56b8fac314ad 2023-03-28 20:54:39,709 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287714e3fee57a42a8806316c9f1a972eb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287714e3fee57a42a8806316c9f1a972eb 2023-03-28 20:54:39,710 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032878e7be6a32534025a04c59e6ff6a1d4a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032878e7be6a32534025a04c59e6ff6a1d4a 2023-03-28 20:54:39,711 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287a0bfd309e2244e0bf9d22fe9c83d7ec to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287a0bfd309e2244e0bf9d22fe9c83d7ec 2023-03-28 20:54:39,712 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287cd3e7fc780d4ac6b1f39ce7aad5df89 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287cd3e7fc780d4ac6b1f39ce7aad5df89 2023-03-28 20:54:39,713 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032888a20aae29bf47e59b913d147f4253af to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032888a20aae29bf47e59b913d147f4253af 2023-03-28 20:54:39,715 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ad331ebd14da4c7a8fa100f70383c450 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ad331ebd14da4c7a8fa100f70383c450 2023-03-28 20:54:39,716 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328c4141c6c6d9a498cb2a0558ea4b8e142 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328c4141c6c6d9a498cb2a0558ea4b8e142 2023-03-28 20:54:39,717 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d05568d8558a47ed8f5e4c1e86aa4ae8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d05568d8558a47ed8f5e4c1e86aa4ae8 2023-03-28 20:54:39,718 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328dbc8a1327e4841db9e36ed5fd077b4f7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328dbc8a1327e4841db9e36ed5fd077b4f7 2023-03-28 20:54:39,719 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ee64c534d2504ae29dab4f580d7f9aab to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ee64c534d2504ae29dab4f580d7f9aab 2023-03-28 20:54:39,720 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ff2179239da5411380cd08b4396ef1bd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ff2179239da5411380cd08b4396ef1bd 2023-03-28 20:54:39,720 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:54:39,722 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,724 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:54:39,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:54:39,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:54:39,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036879727"}]},"ts":"9223372036854775807"} 2023-03-28 20:54:39,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:54:39,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 220e271bfe98d8e1f138571bcf50284c, NAME => 'TestAcidGuarantees,,1680036853424.220e271bfe98d8e1f138571bcf50284c.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:54:39,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:54:39,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036879728"}]},"ts":"9223372036854775807"} 2023-03-28 20:54:39,730 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:54:39,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-03-28 20:54:39,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-28 20:54:39,767 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-28 20:54:39,780 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231 (was 230) - Thread LEAK? -, OpenFileDescriptor=576 (was 551) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 368), ProcessCount=171 (was 173), AvailableMemoryMB=6788 (was 6907) 2023-03-28 20:54:39,791 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=231, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=171, AvailableMemoryMB=6787 2023-03-28 20:54:39,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:54:39,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:54:39,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:54:39,795 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:54:39,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-28 20:54:39,796 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:54:39,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:39,797 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:39,797 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 empty. 2023-03-28 20:54:39,798 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:39,798 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:54:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:40,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:40,205 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:54:40,207 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 16a0b6726aa26da22f14d3e4178de945, NAME => 'TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:54:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 16a0b6726aa26da22f14d3e4178de945, disabling compactions & flushes 2023-03-28 20:54:40,612 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. after waiting 0 ms 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,612 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:40,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:54:40,616 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036880616"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036880616"}]},"ts":"1680036880616"} 2023-03-28 20:54:40,617 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:54:40,618 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:54:40,618 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036880618"}]},"ts":"1680036880618"} 2023-03-28 20:54:40,618 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:54:40,621 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, ASSIGN}] 2023-03-28 20:54:40,622 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, ASSIGN 2023-03-28 20:54:40,623 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:54:40,774 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=16a0b6726aa26da22f14d3e4178de945, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:40,774 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036880774"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036880774"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036880774"}]},"ts":"1680036880774"} 2023-03-28 20:54:40,776 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:54:40,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:40,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 16a0b6726aa26da22f14d3e4178de945, NAME => 'TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:54:40,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:54:40,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,932 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,933 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A 2023-03-28 20:54:40,933 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A 2023-03-28 20:54:40,934 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:40,934 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a0b6726aa26da22f14d3e4178de945 columnFamilyName A 2023-03-28 20:54:40,934 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(310): Store=16a0b6726aa26da22f14d3e4178de945/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:40,935 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,935 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B 2023-03-28 20:54:40,935 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B 2023-03-28 20:54:40,936 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:40,936 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a0b6726aa26da22f14d3e4178de945 columnFamilyName B 2023-03-28 20:54:40,936 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(310): Store=16a0b6726aa26da22f14d3e4178de945/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:40,936 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,937 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C 2023-03-28 20:54:40,937 DEBUG [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C 2023-03-28 20:54:40,937 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:54:40,937 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16a0b6726aa26da22f14d3e4178de945 columnFamilyName C 2023-03-28 20:54:40,938 INFO [StoreOpener-16a0b6726aa26da22f14d3e4178de945-1] regionserver.HStore(310): Store=16a0b6726aa26da22f14d3e4178de945/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:54:40,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:54:40,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:40,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:54:40,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 16a0b6726aa26da22f14d3e4178de945; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69997011, jitterRate=0.04303674399852753}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:54:40,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:40,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., pid=30, masterSystemTime=1680036880927 2023-03-28 20:54:40,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:40,946 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=16a0b6726aa26da22f14d3e4178de945, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:40,946 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036880946"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036880946"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036880946"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036880946"}]},"ts":"1680036880946"} 2023-03-28 20:54:40,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-28 20:54:40,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 in 171 msec 2023-03-28 20:54:40,949 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-28 20:54:40,949 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, ASSIGN in 327 msec 2023-03-28 20:54:40,950 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:54:40,950 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036880950"}]},"ts":"1680036880950"} 2023-03-28 20:54:40,951 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:54:40,954 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:54:40,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-03-28 20:54:41,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-28 20:54:41,900 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-28 20:54:41,901 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,906 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,907 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,908 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,909 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:54:41,911 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34092, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:54:41,912 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,917 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,918 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,922 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,923 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,926 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,926 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x3562c7bf to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,930 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b39d5b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,930 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x6705d115 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,933 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44f9bd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,934 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,938 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,938 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,942 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,942 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,948 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,948 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x2b4fa5d3 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,953 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15186778, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,953 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x45de5ad8 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:54:41,959 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e90e8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:54:41,969 DEBUG [hconnection-0x49a38823-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,970 DEBUG [hconnection-0x7b57f255-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,970 DEBUG [hconnection-0x5ef9a67d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,971 DEBUG [hconnection-0x2f40f6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,971 DEBUG [hconnection-0x670edb4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,972 DEBUG [hconnection-0x5da11e45-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,972 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,972 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46532, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,973 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46560, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:41,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,976 DEBUG [hconnection-0x55d3ba9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,976 DEBUG [hconnection-0x50ec2307-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,976 DEBUG [hconnection-0x483d971b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,977 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46598, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,977 DEBUG [hconnection-0x25e03b7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:54:41,985 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46556, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,985 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,985 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46546, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,987 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:41,987 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:54:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:42,011 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:42,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:42,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:42,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:42,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:42,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:42,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:42,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:42,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:42,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:42,037 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f19198[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:42,039 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:42,039 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,047 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:42,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:42,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:42,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:42,052 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:42,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:42,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:42,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:42,053 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:42,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:42,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,061 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,062 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:42,062 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:42,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:42,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,162 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:42,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:42,166 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:42,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:42,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:42,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,168 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,168 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:42,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:42,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:42,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:42,180 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:42,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:42,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:42,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:42,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:42,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d32f19[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:42,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:42,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:42,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:42,191 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,191 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:42,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:42,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:42,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:42,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:42,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:42,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:42,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:42,200 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,200 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,201 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,203 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:42,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:42,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,207 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:42,207 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:42,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:42,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,307 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:42,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:42,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:42,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:42,311 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:42,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:42,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:42,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:42,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:42,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:42,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:42,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:42,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:42,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:42,319 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,319 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:42,319 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:42,319 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:42,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:42,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:42,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:42,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,322 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:42,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:42,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164d2956[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:42,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,326 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:42,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:42,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:42,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:42,331 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:42,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:42,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:42,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:42,331 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:42,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,338 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:42,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:42,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:42,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,339 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:42,339 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,439 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:42,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:42,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:42,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:42,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:42,442 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:42,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:42,442 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:42,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,444 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:42,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:42,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:42,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:42,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:42,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:42,448 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/7fb9089dd4e64e1f8cea629027635f1a 2023-03-28 20:54:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:42,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6960dd26[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:42,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:42,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:42,456 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,456 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:42,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:42,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,461 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c3d4439884d54459934a6c8ba78c6707 2023-03-28 20:54:42,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:42,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:42,465 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:42,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:42,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:42,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:42,465 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:42,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:42,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:42,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,474 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:42,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:42,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:42,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:42,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:42,478 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:42,478 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:42,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ecd433ba2beb4cafb151cbc42774a60b 2023-03-28 20:54:42,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/7fb9089dd4e64e1f8cea629027635f1a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a 2023-03-28 20:54:42,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a, entries=150, sequenceid=14, filesize=11.6 K 2023-03-28 20:54:42,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c3d4439884d54459934a6c8ba78c6707 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707 2023-03-28 20:54:42,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707, entries=150, sequenceid=14, filesize=11.6 K 2023-03-28 20:54:42,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ecd433ba2beb4cafb151cbc42774a60b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b 2023-03-28 20:54:42,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b, entries=150, sequenceid=14, filesize=11.6 K 2023-03-28 20:54:42,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a0b6726aa26da22f14d3e4178de945 in 501ms, sequenceid=14, compaction requested=false 2023-03-28 20:54:42,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:42,578 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:42,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:42,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:42,585 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:42,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:42,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:42,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:42,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:42,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:42,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:42,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:42,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:42,590 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:42,590 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:42,590 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:42,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:42,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:42,592 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:54:42,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:42,592 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:42,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:42,593 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:42,601 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/074454af429945719ae9cc1c7f8996e6 2023-03-28 20:54:42,634 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/2f800e74bdda44c0923fe7b8f6848c5d 2023-03-28 20:54:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:42,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:42,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036942959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036942959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036942962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036942962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:42,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036942962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,054 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a33258c3bc9d4af69eaaf9c8cb0e5100 2023-03-28 20:54:43,064 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/074454af429945719ae9cc1c7f8996e6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6 2023-03-28 20:54:43,070 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6, entries=150, sequenceid=33, filesize=11.6 K 2023-03-28 20:54:43,070 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/2f800e74bdda44c0923fe7b8f6848c5d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d 2023-03-28 20:54:43,074 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d, entries=150, sequenceid=33, filesize=11.6 K 2023-03-28 20:54:43,075 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a33258c3bc9d4af69eaaf9c8cb0e5100 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100 2023-03-28 20:54:43,078 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100, entries=150, sequenceid=33, filesize=11.6 K 2023-03-28 20:54:43,079 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 16a0b6726aa26da22f14d3e4178de945 in 487ms, sequenceid=33, compaction requested=false 2023-03-28 20:54:43,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:43,079 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:43,081 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,081 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:43,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7ecc3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:43,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:43,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:43,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:43,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:43,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:43,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:43,089 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:43,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:43,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:43,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:43,089 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:43,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:43,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:43,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:43,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:43,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:43,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:43,098 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:43,098 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:43,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:43,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:43,198 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:43,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:43,202 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:43,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:43,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:43,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:43,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:43,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:43,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:43,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:43,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:43,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:43,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:43,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:43,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:43,205 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:43,205 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:43,206 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:43,206 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,213 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/0de23fb71ece4734855b9481f78abb50 2023-03-28 20:54:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:43,264 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:43,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036943274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036943275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036943275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036943275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036943276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036943378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036943379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036943379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036943379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036943379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036943582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036943582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036943582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036943583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036943583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,629 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/2d6a7e534bdf4547834726088499f29c 2023-03-28 20:54:43,654 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a808d7c8942b4f17a1c57ae05daeb358 2023-03-28 20:54:43,660 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/0de23fb71ece4734855b9481f78abb50 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50 2023-03-28 20:54:43,663 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50, entries=150, sequenceid=47, filesize=11.6 K 2023-03-28 20:54:43,663 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-28 20:54:43,664 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/2d6a7e534bdf4547834726088499f29c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c 2023-03-28 20:54:43,668 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c, entries=150, sequenceid=47, filesize=11.6 K 2023-03-28 20:54:43,669 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a808d7c8942b4f17a1c57ae05daeb358 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358 2023-03-28 20:54:43,672 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358, entries=150, sequenceid=47, filesize=11.6 K 2023-03-28 20:54:43,673 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 16a0b6726aa26da22f14d3e4178de945 in 467ms, sequenceid=47, compaction requested=true 2023-03-28 20:54:43,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:43,673 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:43,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:43,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:43,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:43,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:43,676 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc3a043[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:43,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:43,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,680 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,680 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:43,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:43,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:43,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:43,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:43,686 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:43,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:43,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:43,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:43,686 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:43,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:43,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:43,688 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:43,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:43,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:43,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:43,693 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:43,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:43,694 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:43,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:43,694 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:43,794 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:43,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:43,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:43,797 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:43,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:43,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:43,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:43,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:43,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:43,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:43,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:43,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:43,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:43,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:43,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:43,801 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:43,802 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:43,809 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/96a7423b2b3a43d790a154f097d1637c 2023-03-28 20:54:43,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:43,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:43,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036943891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036943892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036943893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036943893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036943894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036943995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036943996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036943996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036943996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:43,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036943997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036944198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036944199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036944199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036944199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036944200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,219 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/18ed36948e4e4f12aaf14c612cb09310 2023-03-28 20:54:44,230 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/8cd91d32d3d24944955b233fb5e01eff 2023-03-28 20:54:44,237 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/96a7423b2b3a43d790a154f097d1637c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c 2023-03-28 20:54:44,241 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c, entries=150, sequenceid=65, filesize=11.6 K 2023-03-28 20:54:44,242 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/18ed36948e4e4f12aaf14c612cb09310 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310 2023-03-28 20:54:44,264 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310, entries=150, sequenceid=65, filesize=11.6 K 2023-03-28 20:54:44,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/8cd91d32d3d24944955b233fb5e01eff as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff 2023-03-28 20:54:44,270 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff, entries=150, sequenceid=65, filesize=11.6 K 2023-03-28 20:54:44,271 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 16a0b6726aa26da22f14d3e4178de945 in 469ms, sequenceid=65, compaction requested=true 2023-03-28 20:54:44,271 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:44,271 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:44,271 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:44,271 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:44,271 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:44,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633b2d14[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:44,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:44,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:44,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:44,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:44,283 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:44,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:44,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:44,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:44,283 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:44,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:44,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:44,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:44,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:44,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:44,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:44,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:44,303 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:44,303 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:44,404 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:44,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:44,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:44,412 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:44,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,414 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:44,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:44,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:44,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:44,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:44,420 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:44,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:44,430 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/e4865dba7832479e9639a09bddc4a6a8 2023-03-28 20:54:44,459 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a33b5849eec84895bbcdd5db6802455e 2023-03-28 20:54:44,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:44,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:44,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036944517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036944518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036944519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036944519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036944520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036944621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036944623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036944624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036944624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036944624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036944825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036944826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036944826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036944827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036944827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,882 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a3e97218879244cdb6d3cb3a1e60fdf0 2023-03-28 20:54:44,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/e4865dba7832479e9639a09bddc4a6a8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8 2023-03-28 20:54:44,890 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8, entries=150, sequenceid=79, filesize=11.6 K 2023-03-28 20:54:44,891 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a33b5849eec84895bbcdd5db6802455e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e 2023-03-28 20:54:44,894 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e, entries=150, sequenceid=79, filesize=11.6 K 2023-03-28 20:54:44,895 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a3e97218879244cdb6d3cb3a1e60fdf0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0 2023-03-28 20:54:44,899 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0, entries=150, sequenceid=79, filesize=11.6 K 2023-03-28 20:54:44,900 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 16a0b6726aa26da22f14d3e4178de945 in 480ms, sequenceid=79, compaction requested=true 2023-03-28 20:54:44,900 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:44,900 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:44,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:44,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:44,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:44,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:44,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d45b8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:44,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:44,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,905 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:44,905 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:44,905 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:44,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:44,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:44,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:44,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:44,910 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:44,910 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:44,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:44,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:44,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:44,914 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:44,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:44,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:44,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:44,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:44,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:44,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:44,917 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:44,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:44,917 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:44,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,013 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:54:45,017 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:45,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:45,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:45,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:45,024 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:45,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:45,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:45,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:45,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:45,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,027 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,027 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:45,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:45,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:45,044 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:45,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,055 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/7a4a05e70ded4b5ba771e4c8f2e12b2c 2023-03-28 20:54:45,067 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e8aad2e087844eaab1eb1a9301b94f02 2023-03-28 20:54:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:45,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:45,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036945136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036945139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036945140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036945140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036945141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036945241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036945244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036945244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036945244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036945244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036945443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036945446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036945447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036945448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036945448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,478 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4add4fa809964ce3abc3b30f9d1ee4d6 2023-03-28 20:54:45,483 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/7a4a05e70ded4b5ba771e4c8f2e12b2c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c 2023-03-28 20:54:45,487 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c, entries=150, sequenceid=97, filesize=11.6 K 2023-03-28 20:54:45,488 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e8aad2e087844eaab1eb1a9301b94f02 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02 2023-03-28 20:54:45,493 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02, entries=150, sequenceid=97, filesize=11.6 K 2023-03-28 20:54:45,494 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4add4fa809964ce3abc3b30f9d1ee4d6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6 2023-03-28 20:54:45,498 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6, entries=150, sequenceid=97, filesize=11.6 K 2023-03-28 20:54:45,499 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 16a0b6726aa26da22f14d3e4178de945 in 455ms, sequenceid=97, compaction requested=true 2023-03-28 20:54:45,499 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:45,499 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,501 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30703104[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:45,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:45,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:45,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:45,506 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:45,509 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:45,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:45,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:45,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:45,509 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:45,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:45,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,511 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:45,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,511 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,513 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,514 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:45,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:45,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:45,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:45,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,515 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:45,515 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:45,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:45,615 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:45,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:45,619 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:45,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:45,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:45,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:45,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,622 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:45,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:45,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:45,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:45,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:45,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:45,624 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:45,624 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:45,750 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:45,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036945762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036945762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036945763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036945765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036945766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036945867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036945867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036945867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036945868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:45,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:45,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036945870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,032 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a6b8fee1168f4b20bb7df180f72a910a 2023-03-28 20:54:46,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036946069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036946069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036946071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036946074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036946074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036946374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036946374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036946374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036946377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036946377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/ac457b1970ca4a94aa650980c797985b 2023-03-28 20:54:46,456 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4a1494b2a5c84c01a84165b85f3c43d1 2023-03-28 20:54:46,461 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a6b8fee1168f4b20bb7df180f72a910a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a 2023-03-28 20:54:46,465 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a, entries=150, sequenceid=112, filesize=11.6 K 2023-03-28 20:54:46,466 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/ac457b1970ca4a94aa650980c797985b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b 2023-03-28 20:54:46,470 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b, entries=150, sequenceid=112, filesize=11.6 K 2023-03-28 20:54:46,471 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4a1494b2a5c84c01a84165b85f3c43d1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1 2023-03-28 20:54:46,477 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1, entries=150, sequenceid=112, filesize=11.6 K 2023-03-28 20:54:46,477 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 16a0b6726aa26da22f14d3e4178de945 in 853ms, sequenceid=112, compaction requested=true 2023-03-28 20:54:46,478 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:46,478 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:46,479 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,479 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31304ef1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:46,481 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:46,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:46,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:46,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:46,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:46,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:46,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:46,488 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:46,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:46,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:46,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:46,488 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:46,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,492 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:46,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:46,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,498 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:46,498 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:46,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,598 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:46,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:46,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:46,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:46,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:46,602 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:46,602 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,602 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:46,602 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:46,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,604 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:46,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:46,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:46,606 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:46,607 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,615 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/85cd5a1b6bc44c72b8feb0ae85da1a06 2023-03-28 20:54:46,625 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c8edf60502374289a61428c2a1e2790d 2023-03-28 20:54:46,641 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/cb1a1b9da42b471d9731a3d53d2dc1af 2023-03-28 20:54:46,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/85cd5a1b6bc44c72b8feb0ae85da1a06 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06 2023-03-28 20:54:46,648 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06, entries=150, sequenceid=129, filesize=11.6 K 2023-03-28 20:54:46,649 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c8edf60502374289a61428c2a1e2790d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d 2023-03-28 20:54:46,652 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d, entries=150, sequenceid=129, filesize=11.6 K 2023-03-28 20:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/cb1a1b9da42b471d9731a3d53d2dc1af as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af 2023-03-28 20:54:46,656 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af, entries=150, sequenceid=129, filesize=11.6 K 2023-03-28 20:54:46,657 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 16a0b6726aa26da22f14d3e4178de945 in 51ms, sequenceid=129, compaction requested=true 2023-03-28 20:54:46,657 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:46,657 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,657 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:46,657 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:46,657 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,657 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:46,657 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd9864e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:46,661 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:46,661 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,662 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:46,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:46,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,664 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:46,664 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:46,667 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:46,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:46,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:46,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:46,667 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:46,668 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,668 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,669 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,671 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:46,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,673 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:46,673 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,773 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:46,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:46,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:46,777 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:46,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:46,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:46,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,779 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,779 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:46,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:46,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:46,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:46,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:46,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:46,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:46,781 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:46,783 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dcf7208[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:46,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:46,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,787 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,787 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:46,789 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:46,792 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:46,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:46,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:46,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:46,792 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:46,793 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,793 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,794 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,794 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,798 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,798 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:46,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:46,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,799 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:46,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:46,799 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:46,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:46,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:46,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:46,899 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:46,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:46,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:46,903 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:46,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:46,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:46,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:46,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:46,908 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,908 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:46,908 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:46,908 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:46,910 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036946904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036946904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036946905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8696a4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:46,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:46,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036946907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036946907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,913 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:46,914 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,914 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:46,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:46,919 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:46,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:46,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:46,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:46,919 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:46,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,921 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:46,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:46,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,923 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,925 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:46,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:46,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:46,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:46,926 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:46,926 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:46,926 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:46,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036947011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036947011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036947012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036947013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036947013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,026 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:47,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:47,029 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:47,029 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,029 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:47,029 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:47,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,035 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:47,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:47,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,043 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,043 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff20c03[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:47,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:47,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,045 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:47,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:47,047 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:47,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:47,049 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:47,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:47,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:47,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:47,050 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:47,056 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,056 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,056 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,056 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:47,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,060 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:47,060 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,160 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:47,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:47,163 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:47,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:47,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:47,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:47,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:47,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:47,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:47,168 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:47,170 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28de0c45[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:47,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:47,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,174 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:47,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:47,176 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:47,179 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:47,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:47,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:47,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:47,179 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:47,181 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,181 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,181 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,183 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:47,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:47,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,186 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:47,186 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:47,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036947215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036947216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036947216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036947217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036947217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,286 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:47,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:47,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:47,290 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,293 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,293 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,293 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,293 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/39b46f42b1d041a89cb0c1f4efcb0b9d 2023-03-28 20:54:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:47,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:47,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:47,298 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,298 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:47,298 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:47,298 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:47,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:47,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e976464[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:47,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:47,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:47,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:47,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:47,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,305 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a03a41b2d1c14d86923dfe714ec3ab05 2023-03-28 20:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:47,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:47,309 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:47,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:47,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:47,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:47,309 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:47,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:47,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,317 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,318 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:47,318 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:47,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/dc432f52da5743d0a1926f5c0afc9cc3 2023-03-28 20:54:47,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/39b46f42b1d041a89cb0c1f4efcb0b9d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d 2023-03-28 20:54:47,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d, entries=300, sequenceid=142, filesize=18.8 K 2023-03-28 20:54:47,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a03a41b2d1c14d86923dfe714ec3ab05 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05 2023-03-28 20:54:47,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:54:47,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/dc432f52da5743d0a1926f5c0afc9cc3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3 2023-03-28 20:54:47,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3, entries=150, sequenceid=142, filesize=11.8 K 2023-03-28 20:54:47,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 449ms, sequenceid=142, compaction requested=true 2023-03-28 20:54:47,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:47,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:47,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:47,337 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:54:47,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-28 20:54:47,338 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:54:47,341 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 114294 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:54:47,341 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:47,341 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,341 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107124 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:54:47,341 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=111.6 K 2023-03-28 20:54:47,342 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:47,342 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,342 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=104.6 K 2023-03-28 20:54:47,342 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7fb9089dd4e64e1f8cea629027635f1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680036881988 2023-03-28 20:54:47,342 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c3d4439884d54459934a6c8ba78c6707, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680036881988 2023-03-28 20:54:47,343 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 074454af429945719ae9cc1c7f8996e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680036882027 2023-03-28 20:54:47,343 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2f800e74bdda44c0923fe7b8f6848c5d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680036882027 2023-03-28 20:54:47,343 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0de23fb71ece4734855b9481f78abb50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680036882649 2023-03-28 20:54:47,343 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2d6a7e534bdf4547834726088499f29c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680036882649 2023-03-28 20:54:47,344 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 96a7423b2b3a43d790a154f097d1637c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680036883273 2023-03-28 20:54:47,344 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 18ed36948e4e4f12aaf14c612cb09310, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680036883273 2023-03-28 20:54:47,344 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e4865dba7832479e9639a09bddc4a6a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680036883889 2023-03-28 20:54:47,344 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a33b5849eec84895bbcdd5db6802455e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680036883889 2023-03-28 20:54:47,345 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7a4a05e70ded4b5ba771e4c8f2e12b2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680036884512 2023-03-28 20:54:47,345 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e8aad2e087844eaab1eb1a9301b94f02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680036884512 2023-03-28 20:54:47,345 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a6b8fee1168f4b20bb7df180f72a910a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680036885139 2023-03-28 20:54:47,345 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ac457b1970ca4a94aa650980c797985b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680036885139 2023-03-28 20:54:47,346 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 85cd5a1b6bc44c72b8feb0ae85da1a06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680036885757 2023-03-28 20:54:47,346 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c8edf60502374289a61428c2a1e2790d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680036885757 2023-03-28 20:54:47,346 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 39b46f42b1d041a89cb0c1f4efcb0b9d, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886882 2023-03-28 20:54:47,347 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a03a41b2d1c14d86923dfe714ec3ab05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886886 2023-03-28 20:54:47,377 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#242 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:47,377 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#243 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:47,418 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:47,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:47,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:47,421 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:47,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:47,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:47,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,422 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:47,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:47,425 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:47,433 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/33c313adaf4d4a2cbb8cacd854168115 2023-03-28 20:54:47,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/3b9cb6618e024b82ad273925d7472b46 2023-03-28 20:54:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:47,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036947527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036947531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036947532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036947533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036947533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036947634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036947637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036947637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036947638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036947638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,790 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/b22f2d3fba044c85821f940f6cb6cc4c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b22f2d3fba044c85821f940f6cb6cc4c 2023-03-28 20:54:47,796 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into b22f2d3fba044c85821f940f6cb6cc4c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:47,796 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:47,796 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=7, startTime=1680036887337; duration=0sec 2023-03-28 20:54:47,797 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:47,797 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-28 20:54:47,797 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/70e7f91e669a498b99ccaf64617225fc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/70e7f91e669a498b99ccaf64617225fc 2023-03-28 20:54:47,800 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107124 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-28 20:54:47,800 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:47,800 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,800 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=104.6 K 2023-03-28 20:54:47,800 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ecd433ba2beb4cafb151cbc42774a60b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680036881988 2023-03-28 20:54:47,801 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a33258c3bc9d4af69eaaf9c8cb0e5100, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680036882027 2023-03-28 20:54:47,801 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a808d7c8942b4f17a1c57ae05daeb358, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680036882649 2023-03-28 20:54:47,802 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8cd91d32d3d24944955b233fb5e01eff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680036883273 2023-03-28 20:54:47,802 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 70e7f91e669a498b99ccaf64617225fc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:47,802 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:47,802 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=7, startTime=1680036887337; duration=0sec 2023-03-28 20:54:47,802 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a3e97218879244cdb6d3cb3a1e60fdf0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680036883889 2023-03-28 20:54:47,802 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:47,803 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4add4fa809964ce3abc3b30f9d1ee4d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680036884512 2023-03-28 20:54:47,803 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4a1494b2a5c84c01a84165b85f3c43d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680036885139 2023-03-28 20:54:47,803 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cb1a1b9da42b471d9731a3d53d2dc1af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680036885757 2023-03-28 20:54:47,804 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting dc432f52da5743d0a1926f5c0afc9cc3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886886 2023-03-28 20:54:47,821 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#247 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:47,830 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/321b3fb532b142ebb5e7ad7309e80bd8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/321b3fb532b142ebb5e7ad7309e80bd8 2023-03-28 20:54:47,836 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 321b3fb532b142ebb5e7ad7309e80bd8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:47,836 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=7, startTime=1680036887337; duration=0sec 2023-03-28 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:47,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036947838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036947841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036947842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036947842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036947842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,854 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/eb4d75bad6ae45748d3de00132d1198e 2023-03-28 20:54:47,859 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/33c313adaf4d4a2cbb8cacd854168115 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115 2023-03-28 20:54:47,867 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115, entries=150, sequenceid=162, filesize=11.8 K 2023-03-28 20:54:47,869 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/3b9cb6618e024b82ad273925d7472b46 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46 2023-03-28 20:54:47,880 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46, entries=150, sequenceid=162, filesize=11.8 K 2023-03-28 20:54:47,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/eb4d75bad6ae45748d3de00132d1198e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e 2023-03-28 20:54:47,885 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e, entries=150, sequenceid=162, filesize=11.8 K 2023-03-28 20:54:47,886 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 16a0b6726aa26da22f14d3e4178de945 in 461ms, sequenceid=162, compaction requested=false 2023-03-28 20:54:47,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:47,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:47,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,891 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564ce6a4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:47,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:47,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:47,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:47,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:47,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:47,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:47,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:47,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:47,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:47,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:47,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:47,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:47,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:47,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:47,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:47,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:47,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:47,907 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:47,908 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:47,908 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:48,008 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:48,011 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:48,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:48,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:48,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:48,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,013 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:48,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:48,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:48,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:48,015 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:48,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:48,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:48,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:48,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036948160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036948160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036948160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036948161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036948162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036948265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036948267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036948267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036948266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036948267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,427 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a05c2cb16a284420a2f44b5fe96baf57 2023-03-28 20:54:48,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036948468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036948470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036948471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036948472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036948472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036948771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036948773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036948773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036948774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036948775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,840 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/8c856d391c20470195cd0dec2679c48f 2023-03-28 20:54:48,850 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/77c240d652ca49bf978950365b250564 2023-03-28 20:54:48,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a05c2cb16a284420a2f44b5fe96baf57 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57 2023-03-28 20:54:48,857 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57, entries=150, sequenceid=178, filesize=11.8 K 2023-03-28 20:54:48,858 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/8c856d391c20470195cd0dec2679c48f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f 2023-03-28 20:54:48,861 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f, entries=150, sequenceid=178, filesize=11.8 K 2023-03-28 20:54:48,862 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/77c240d652ca49bf978950365b250564 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564 2023-03-28 20:54:48,865 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564, entries=150, sequenceid=178, filesize=11.8 K 2023-03-28 20:54:48,866 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a0b6726aa26da22f14d3e4178de945 in 851ms, sequenceid=178, compaction requested=true 2023-03-28 20:54:48,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:48,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:48,869 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a17a7a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:48,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:48,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:48,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:48,873 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:48,877 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:48,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:48,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:48,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:48,877 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:48,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:48,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:48,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:48,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:48,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:48,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:48,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:48,886 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:48,886 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:48,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:48,986 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:48,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:48,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:48,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:48,990 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:48,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:48,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:48,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:48,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,992 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:48,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:48,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:48,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:48,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:48,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:48,994 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:48,994 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:48,995 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:48,995 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:49,006 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1d1587c7cad944e9b8f8a6b90ab02288 2023-03-28 20:54:49,018 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/1d7bbcf72cea43b9aa625de521137e35 2023-03-28 20:54:49,029 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/f03dc6ae09074ebcac498b265c22fcdb 2023-03-28 20:54:49,034 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1d1587c7cad944e9b8f8a6b90ab02288 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288 2023-03-28 20:54:49,039 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:54:49,039 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/1d7bbcf72cea43b9aa625de521137e35 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35 2023-03-28 20:54:49,043 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:54:49,044 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/f03dc6ae09074ebcac498b265c22fcdb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb 2023-03-28 20:54:49,052 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb, entries=150, sequenceid=197, filesize=11.8 K 2023-03-28 20:54:49,053 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 16a0b6726aa26da22f14d3e4178de945 in 58ms, sequenceid=197, compaction requested=true 2023-03-28 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,055 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c5f72f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,061 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,064 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,064 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,067 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,072 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,072 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,172 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,176 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,177 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,177 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,181 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,181 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,181 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:49,181 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f40937[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,188 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,193 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,195 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,196 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:49,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,201 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,202 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,201 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,202 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:49,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:54:49,283 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:49,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:49,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036949298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,302 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036889294, max timestamp=1680036889300Num uniques 859; 2023-03-28 20:54:49,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036889294, max timestamp=1680036889300Num uniques 859; 2023-03-28 20:54:49,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036889294, max timestamp=1680036889300Num uniques 859; 2023-03-28 20:54:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:54:49,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/b32a4cef629a40578ab74a3aae30fb87 2023-03-28 20:54:49,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,306 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,312 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,312 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,312 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,312 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453ca8cb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,321 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,322 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036949319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,323 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036949322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036949323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036949323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,328 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,329 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,333 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:49,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,333 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036949403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036949424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036949427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036949427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036949427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,434 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,437 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ec78bc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,446 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,446 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,452 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,454 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,454 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:49,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,460 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,460 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,461 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,561 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,568 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,569 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,569 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,571 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,574 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,576 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ce97e1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,580 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,582 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,585 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,585 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,596 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,596 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,596 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036949606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036949628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036949631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036949632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036949632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,696 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,700 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,704 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,707 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d278c5e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,710 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,714 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,714 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,716 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,721 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,721 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f26ebc9c43094b048353941a7c0642b3 2023-03-28 20:54:49,824 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,828 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,832 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dfd1bc1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,841 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,844 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,844 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,846 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,851 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,851 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,852 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,852 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036949909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036949932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036949934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036949937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036949937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,951 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:49,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:49,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:49,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:49,954 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:49,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:49,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:49,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,957 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:49,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:49,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:49,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:49,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:49,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:49,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:49,958 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,958 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:49,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:49,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:49,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:49,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:49,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:49,959 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:49,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4950a35a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:49,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:49,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,963 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:49,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:49,964 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:49,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:49,966 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:49,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:49,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:49,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:49,967 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:49,969 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,969 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:49,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,969 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:49,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,971 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:49,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:49,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:49,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:49,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:49,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:49,976 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:49,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,977 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:49,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:49,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,077 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:50,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:50,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:50,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:50,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:50,080 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:50,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:50,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:50,081 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,081 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:50,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:50,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:50,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:50,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:50,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,086 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,087 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:50,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:50,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec540e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:50,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:50,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:50,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:50,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:50,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:50,094 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:50,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:50,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:50,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:50,094 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:50,096 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,096 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,096 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:50,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:50,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,101 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:50,101 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:50,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:50,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/202ae780ee284f9ca13cf8111a6ee31b 2023-03-28 20:54:50,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/b32a4cef629a40578ab74a3aae30fb87 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87 2023-03-28 20:54:50,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87, entries=150, sequenceid=209, filesize=11.8 K 2023-03-28 20:54:50,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f26ebc9c43094b048353941a7c0642b3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3 2023-03-28 20:54:50,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3, entries=150, sequenceid=209, filesize=11.8 K 2023-03-28 20:54:50,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/202ae780ee284f9ca13cf8111a6ee31b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b 2023-03-28 20:54:50,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b, entries=150, sequenceid=209, filesize=11.8 K 2023-03-28 20:54:50,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 16a0b6726aa26da22f14d3e4178de945 in 879ms, sequenceid=209, compaction requested=true 2023-03-28 20:54:50,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,162 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:50,162 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:50,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,163 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:50,163 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:50,164 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,164 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:50,164 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/70e7f91e669a498b99ccaf64617225fc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=59.1 K 2023-03-28 20:54:50,164 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:50,164 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,164 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b22f2d3fba044c85821f940f6cb6cc4c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=59.1 K 2023-03-28 20:54:50,164 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 70e7f91e669a498b99ccaf64617225fc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886886 2023-03-28 20:54:50,164 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b22f2d3fba044c85821f940f6cb6cc4c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886886 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 33c313adaf4d4a2cbb8cacd854168115, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036886899 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3b9cb6618e024b82ad273925d7472b46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036886899 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a05c2cb16a284420a2f44b5fe96baf57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680036887531 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8c856d391c20470195cd0dec2679c48f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680036887531 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1d1587c7cad944e9b8f8a6b90ab02288, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036888152 2023-03-28 20:54:50,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1d7bbcf72cea43b9aa625de521137e35, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036888152 2023-03-28 20:54:50,166 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b32a4cef629a40578ab74a3aae30fb87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,166 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f26ebc9c43094b048353941a7c0642b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,186 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#258 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,186 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,201 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:50,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:50,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:50,205 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:50,205 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:50,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:50,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:50,209 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/26a47550a72c4e5c9fba17d36a24bf6b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/26a47550a72c4e5c9fba17d36a24bf6b 2023-03-28 20:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:50,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:50,210 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:50,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,211 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:50,211 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,217 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 26a47550a72c4e5c9fba17d36a24bf6b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,217 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,218 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=11, startTime=1680036890162; duration=0sec 2023-03-28 20:54:50,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:50,220 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:50,221 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:50,221 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,221 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/321b3fb532b142ebb5e7ad7309e80bd8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=59.1 K 2023-03-28 20:54:50,221 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 321b3fb532b142ebb5e7ad7309e80bd8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680036886886 2023-03-28 20:54:50,221 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting eb4d75bad6ae45748d3de00132d1198e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680036886899 2023-03-28 20:54:50,222 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 77c240d652ca49bf978950365b250564, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680036887531 2023-03-28 20:54:50,222 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f03dc6ae09074ebcac498b265c22fcdb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680036888152 2023-03-28 20:54:50,222 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 202ae780ee284f9ca13cf8111a6ee31b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,246 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,262 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/277dd6fec2c74697a29e18726998b304 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/277dd6fec2c74697a29e18726998b304 2023-03-28 20:54:50,266 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 277dd6fec2c74697a29e18726998b304(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,266 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,267 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=11, startTime=1680036890162; duration=0sec 2023-03-28 20:54:50,267 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:50,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:50,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036950434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036950438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036950441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036950442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036950446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46520 deadline: 1680036950536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,607 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/cde54d8920dd4620add4e285209fce34 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/cde54d8920dd4620add4e285209fce34 2023-03-28 20:54:50,611 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into cde54d8920dd4620add4e285209fce34(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,611 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,611 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=11, startTime=1680036890162; duration=0sec 2023-03-28 20:54:50,612 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,650 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/5f5e040d06904a39920ca6f04ab6c8ad 2023-03-28 20:54:50,659 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/7dd4099b616b4ea9b92295a94c6d87de 2023-03-28 20:54:50,668 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ae44b79039d24dc5bcf2305da4c23f1d 2023-03-28 20:54:50,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/5f5e040d06904a39920ca6f04ab6c8ad as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad 2023-03-28 20:54:50,674 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad, entries=150, sequenceid=244, filesize=11.8 K 2023-03-28 20:54:50,675 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/7dd4099b616b4ea9b92295a94c6d87de as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de 2023-03-28 20:54:50,679 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de, entries=150, sequenceid=244, filesize=11.8 K 2023-03-28 20:54:50,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ae44b79039d24dc5bcf2305da4c23f1d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d 2023-03-28 20:54:50,683 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d, entries=150, sequenceid=244, filesize=11.8 K 2023-03-28 20:54:50,684 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 16a0b6726aa26da22f14d3e4178de945 in 474ms, sequenceid=244, compaction requested=false 2023-03-28 20:54:50,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:50,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55786db1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:50,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:50,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,689 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:50,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:50,691 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:50,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,691 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,691 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:50,694 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:50,694 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:50,694 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:50,694 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:50,694 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:50,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:50,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:50,702 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:50,702 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:50,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:50,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:54:50,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:50,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:50,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:50,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:50,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:50,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:50,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/8de3c65e6cc447109ff47375dcb04e82 2023-03-28 20:54:50,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/fade0e9c451b41829efa13d749b4d88f 2023-03-28 20:54:50,802 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:50,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:50,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:50,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:50,812 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:50,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:50,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:50,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/638c3396c5a74c17a7e7d44c174ab9b5 2023-03-28 20:54:50,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:50,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:50,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036890778, max timestamp=1680036890815Num uniques 859; 2023-03-28 20:54:50,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:50,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036890778, max timestamp=1680036890815Num uniques 859; 2023-03-28 20:54:50,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:50,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036890778, max timestamp=1680036890815Num uniques 859; 2023-03-28 20:54:50,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:50,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:50,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:50,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:50,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/8de3c65e6cc447109ff47375dcb04e82 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82 2023-03-28 20:54:50,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82, entries=150, sequenceid=258, filesize=11.8 K 2023-03-28 20:54:50,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:50,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe47cc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:50,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:50,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/fade0e9c451b41829efa13d749b4d88f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f 2023-03-28 20:54:50,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:50,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:50,830 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:50,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:50,832 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:50,832 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:50,832 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:50,832 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:50,832 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:50,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f, entries=150, sequenceid=258, filesize=11.8 K 2023-03-28 20:54:50,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/638c3396c5a74c17a7e7d44c174ab9b5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5 2023-03-28 20:54:50,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,836 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,840 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,840 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5, entries=150, sequenceid=258, filesize=11.8 K 2023-03-28 20:54:50,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:50,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:50,841 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:50,841 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:50,840 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,842 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=46.96 KB/48090 for 16a0b6726aa26da22f14d3e4178de945 in 101ms, sequenceid=258, compaction requested=true 2023-03-28 20:54:50,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:50,842 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,842 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:50,842 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:50,842 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:50,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,843 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:50,844 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:50,844 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:50,844 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:50,844 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,844 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/26a47550a72c4e5c9fba17d36a24bf6b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=35.8 K 2023-03-28 20:54:50,844 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,844 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/cde54d8920dd4620add4e285209fce34, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=35.8 K 2023-03-28 20:54:50,845 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cde54d8920dd4620add4e285209fce34, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,845 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 26a47550a72c4e5c9fba17d36a24bf6b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,845 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5f5e040d06904a39920ca6f04ab6c8ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680036889307 2023-03-28 20:54:50,845 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7dd4099b616b4ea9b92295a94c6d87de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680036889307 2023-03-28 20:54:50,845 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8de3c65e6cc447109ff47375dcb04e82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:50,846 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fade0e9c451b41829efa13d749b4d88f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:50,857 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,859 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/8fae27fe06104cf78c1e2203d2a90754 2023-03-28 20:54:50,879 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/45d4db048e6a4c49ab3ae438a122b489 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/45d4db048e6a4c49ab3ae438a122b489 2023-03-28 20:54:50,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/09c5dfbe39bf4bc495344ab4182e09b3 2023-03-28 20:54:50,882 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/71dad285676844ebb7c2dfd91be1599d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/71dad285676844ebb7c2dfd91be1599d 2023-03-28 20:54:50,885 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 45d4db048e6a4c49ab3ae438a122b489(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,885 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036890842; duration=0sec 2023-03-28 20:54:50,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:50,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:50,886 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:50,886 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:50,886 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,887 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/277dd6fec2c74697a29e18726998b304, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=35.8 K 2023-03-28 20:54:50,887 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 277dd6fec2c74697a29e18726998b304, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680036889280 2023-03-28 20:54:50,888 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ae44b79039d24dc5bcf2305da4c23f1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680036889307 2023-03-28 20:54:50,888 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 71dad285676844ebb7c2dfd91be1599d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,888 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,888 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036890842; duration=0sec 2023-03-28 20:54:50,889 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,889 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 638c3396c5a74c17a7e7d44c174ab9b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:50,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/50eb1a3e5fa04e809ab4ab481a85547f 2023-03-28 20:54:50,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/8fae27fe06104cf78c1e2203d2a90754 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754 2023-03-28 20:54:50,905 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:50,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754, entries=200, sequenceid=283, filesize=14.3 K 2023-03-28 20:54:50,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/09c5dfbe39bf4bc495344ab4182e09b3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3 2023-03-28 20:54:50,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3, entries=150, sequenceid=283, filesize=11.9 K 2023-03-28 20:54:50,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/50eb1a3e5fa04e809ab4ab481a85547f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f 2023-03-28 20:54:50,918 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/25fd3c979fa249619ff9681f1691d8b7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/25fd3c979fa249619ff9681f1691d8b7 2023-03-28 20:54:50,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f, entries=150, sequenceid=283, filesize=11.9 K 2023-03-28 20:54:50,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 77ms, sequenceid=283, compaction requested=false 2023-03-28 20:54:50,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,922 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 25fd3c979fa249619ff9681f1691d8b7(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:50,922 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:50,922 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=13, startTime=1680036890843; duration=0sec 2023-03-28 20:54:50,922 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:50,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-28 20:54:50,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:50,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ca2d5ca30dc44af4b89c9979bf8be287 2023-03-28 20:54:50,941 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:50,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:50,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:50,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036950951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:50,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:50,956 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c51aef9d703e46aeb3dd7ba653c81926 2023-03-28 20:54:50,961 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,961 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:50,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:50,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:50,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:50,965 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:50,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf870fa[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:50,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:50,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,974 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:50,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:50,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:50,974 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,974 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:50,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:50,979 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:50,979 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:50,979 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:50,979 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:50,979 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:50,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:50,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ea52078f81964068a8fc28be676cf930 2023-03-28 20:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:50,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:50,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:50,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:50,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:50,987 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:50,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:50,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:50,988 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:50,988 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:50,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ca2d5ca30dc44af4b89c9979bf8be287 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287 2023-03-28 20:54:50,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287, entries=200, sequenceid=307, filesize=14.3 K 2023-03-28 20:54:50,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c51aef9d703e46aeb3dd7ba653c81926 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926 2023-03-28 20:54:50,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926, entries=150, sequenceid=307, filesize=11.9 K 2023-03-28 20:54:51,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ea52078f81964068a8fc28be676cf930 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930 2023-03-28 20:54:51,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930, entries=150, sequenceid=307, filesize=11.9 K 2023-03-28 20:54:51,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 16a0b6726aa26da22f14d3e4178de945 in 81ms, sequenceid=307, compaction requested=true 2023-03-28 20:54:51,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:51,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:51,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:51,005 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:51,005 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:51,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:51,006 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:51,006 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37038 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:51,006 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:51,006 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:51,006 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,006 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,006 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/45d4db048e6a4c49ab3ae438a122b489, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=40.9 K 2023-03-28 20:54:51,006 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/71dad285676844ebb7c2dfd91be1599d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.2 K 2023-03-28 20:54:51,007 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 45d4db048e6a4c49ab3ae438a122b489, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:51,007 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 71dad285676844ebb7c2dfd91be1599d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:51,007 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8fae27fe06104cf78c1e2203d2a90754, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680036890815 2023-03-28 20:54:51,007 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 09c5dfbe39bf4bc495344ab4182e09b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680036890815 2023-03-28 20:54:51,008 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ca2d5ca30dc44af4b89c9979bf8be287, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890906 2023-03-28 20:54:51,008 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c51aef9d703e46aeb3dd7ba653c81926, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890909 2023-03-28 20:54:51,019 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:51,019 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:51,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:51,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:51,088 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,091 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,091 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,091 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,091 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad21f4f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,102 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,106 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,106 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,106 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,106 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,106 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,113 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,114 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,114 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036951117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,214 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,217 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,218 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,218 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,221 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,221 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,221 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,221 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036951221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,224 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1e441[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,227 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,228 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,231 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,231 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,243 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,243 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,343 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,348 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,349 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,350 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,367 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e930a30[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,372 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,377 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,377 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,382 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,382 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036951425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/33e397ba6b9f42dc8a87d3542af71334 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33e397ba6b9f42dc8a87d3542af71334 2023-03-28 20:54:51,439 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 33e397ba6b9f42dc8a87d3542af71334(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:51,440 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:51,440 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036891005; duration=0sec 2023-03-28 20:54:51,440 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:51,440 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:51,441 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37038 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:51,441 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:51,441 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,441 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/25fd3c979fa249619ff9681f1691d8b7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.2 K 2023-03-28 20:54:51,441 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/9f3552a5b46140b583a5b0ce8c28b3bb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/9f3552a5b46140b583a5b0ce8c28b3bb 2023-03-28 20:54:51,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 25fd3c979fa249619ff9681f1691d8b7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680036890425 2023-03-28 20:54:51,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 50eb1a3e5fa04e809ab4ab481a85547f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680036890815 2023-03-28 20:54:51,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ea52078f81964068a8fc28be676cf930, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890909 2023-03-28 20:54:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036951443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,446 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 9f3552a5b46140b583a5b0ce8c28b3bb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:51,446 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:51,446 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036891005; duration=0sec 2023-03-28 20:54:51,446 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:51,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036951448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036951448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,450 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:51,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036951456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,458 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/1ca9977218bb4168abd8fd4a65ce0c4a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1ca9977218bb4168abd8fd4a65ce0c4a 2023-03-28 20:54:51,462 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 1ca9977218bb4168abd8fd4a65ce0c4a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:51,462 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:51,462 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=13, startTime=1680036891005; duration=0sec 2023-03-28 20:54:51,462 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:51,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/aa5b82e5d19441c1b1af461909c40587 2023-03-28 20:54:51,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c3672cd8b8a5425c9e87a6cad62fec60 2023-03-28 20:54:51,483 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,486 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,488 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,489 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,493 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3113ccb8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,502 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,502 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,510 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,510 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,510 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,510 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,611 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,614 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,615 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,620 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf185b2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,626 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,628 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,631 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,631 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,633 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,633 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,637 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,637 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036951731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,737 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,742 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,743 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,745 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,745 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,746 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,749 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c18561[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,753 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,760 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,760 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,763 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,763 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,763 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:51,777 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,778 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,878 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:51,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:51,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:51,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:51,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:51,882 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:51,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:51,885 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:51,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:51,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:51,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:51,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:51,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:51,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e55cac3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:51,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/22d7dab36c6d497f9645183397d4882d 2023-03-28 20:54:51,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:51,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/aa5b82e5d19441c1b1af461909c40587 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587 2023-03-28 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:51,897 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:51,897 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:51,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:51,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:51,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587, entries=150, sequenceid=319, filesize=11.9 K 2023-03-28 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c3672cd8b8a5425c9e87a6cad62fec60 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60 2023-03-28 20:54:51,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:51,904 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:51,904 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:51,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:51,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:51,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60, entries=150, sequenceid=319, filesize=11.9 K 2023-03-28 20:54:51,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/22d7dab36c6d497f9645183397d4882d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d 2023-03-28 20:54:51,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d, entries=150, sequenceid=319, filesize=11.9 K 2023-03-28 20:54:51,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 852ms, sequenceid=319, compaction requested=false 2023-03-28 20:54:51,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:52,004 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:52,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:52,007 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:52,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:52,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:52,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:52,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:52,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:52,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:52,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:52,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:52,012 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:52,013 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:52,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:52,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:52,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:52,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036952256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:52,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036952359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,420 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ac1f8bc34b09416cab71a894422f67d5 2023-03-28 20:54:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036952563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,842 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/07e95d0576ab4fa19b084e14c154b89b 2023-03-28 20:54:52,852 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4f6a31938780415d9a61a5d962855a12 2023-03-28 20:54:52,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ac1f8bc34b09416cab71a894422f67d5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5 2023-03-28 20:54:52,861 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5, entries=150, sequenceid=342, filesize=11.9 K 2023-03-28 20:54:52,862 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/07e95d0576ab4fa19b084e14c154b89b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b 2023-03-28 20:54:52,866 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b, entries=150, sequenceid=342, filesize=11.9 K 2023-03-28 20:54:52,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4f6a31938780415d9a61a5d962855a12 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12 2023-03-28 20:54:52,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036952866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,869 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12, entries=150, sequenceid=342, filesize=11.9 K 2023-03-28 20:54:52,870 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 16a0b6726aa26da22f14d3e4178de945 in 858ms, sequenceid=342, compaction requested=true 2023-03-28 20:54:52,870 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:52,870 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:52,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418cbdbd[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:52,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:52,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,875 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:52,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:52,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:52,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:52,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:52,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:52,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:52,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:52,879 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:52,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:52,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:52,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:52,879 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:52,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:52,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:52,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:52,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,893 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:52,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:52,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:52,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:52,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:52,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:52,894 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:52,894 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:52,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:52,995 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:52,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:52,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:52,998 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:52,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:52,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:52,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:52,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:52,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,000 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:53,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:53,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:53,002 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:53,018 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/c988989f510f4f78950bbd54fa37cb12 2023-03-28 20:54:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:53,373 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:53,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036953421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,430 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/ab2bc385243744eabc1f3e1e85b126b9 2023-03-28 20:54:53,439 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3e6909ebc3b0460e87f8003fa9cf0276 2023-03-28 20:54:53,443 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/c988989f510f4f78950bbd54fa37cb12 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12 2023-03-28 20:54:53,446 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12, entries=150, sequenceid=354, filesize=11.9 K 2023-03-28 20:54:53,446 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/ab2bc385243744eabc1f3e1e85b126b9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9 2023-03-28 20:54:53,450 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9, entries=150, sequenceid=354, filesize=11.9 K 2023-03-28 20:54:53,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036953450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,452 DEBUG [Thread-1721] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:53,452 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3e6909ebc3b0460e87f8003fa9cf0276 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276 2023-03-28 20:54:53,456 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276, entries=150, sequenceid=354, filesize=11.9 K 2023-03-28 20:54:53,457 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 455ms, sequenceid=354, compaction requested=true 2023-03-28 20:54:53,457 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:53,457 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:53,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:53,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:53,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:53,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-28 20:54:53,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:53,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:53,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:53,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:53,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:53,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:53,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36276c2b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:53,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:53,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,465 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:53,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:53,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:53,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:53,467 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:53,470 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:53,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:53,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:53,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:53,470 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:53,475 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,475 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,475 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,475 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036953473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036953476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036953476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,479 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:53,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,480 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:53,484 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:53,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036953525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036953577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46598 deadline: 1680036953580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036953580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,584 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:53,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:53,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:53,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:53,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:53,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:53,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:53,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,588 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,589 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:53,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:53,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:53,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:53,591 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:53,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f91d03[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,600 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,601 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:53,601 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:53,601 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:53,602 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:53,604 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:53,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:53,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:53,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:53,604 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:53,606 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,606 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,606 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,608 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:53,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:53,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,610 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,610 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:53,611 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:53,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,711 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:53,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:53,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:53,714 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:53,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:53,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:53,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:53,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:53,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:53,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:53,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:53,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423adc4d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:53,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:53,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:53,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:53,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:53,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,725 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:53,727 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:53,727 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:53,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:53,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:53,728 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:53,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036953728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,736 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:53,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:53,736 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:53,736 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46560 deadline: 1680036953786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036953786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036953786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,836 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:53,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:53,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:53,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:53,840 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:53,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:53,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:53,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,842 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,843 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,843 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:53,844 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,844 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:53,844 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:53,845 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:53,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ffe0e2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:53,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:53,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:53,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:53,852 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:53,855 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:53,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:53,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:53,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:53,855 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:53,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:53,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,857 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:53,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,864 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,865 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:53,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:53,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:53,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,865 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:53,865 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:53,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/56d8417312e9491a9534c4f8cbd41640 2023-03-28 20:54:53,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/cb123333a6414403b924fff1caa1de2a 2023-03-28 20:54:53,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a926e23988144743b476aa2d72e13dbc 2023-03-28 20:54:53,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/56d8417312e9491a9534c4f8cbd41640 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640 2023-03-28 20:54:53,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640, entries=200, sequenceid=375, filesize=14.3 K 2023-03-28 20:54:53,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/cb123333a6414403b924fff1caa1de2a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a 2023-03-28 20:54:53,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a, entries=150, sequenceid=375, filesize=11.9 K 2023-03-28 20:54:53,903 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-28 20:54:53,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a926e23988144743b476aa2d72e13dbc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc 2023-03-28 20:54:53,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc, entries=150, sequenceid=375, filesize=11.9 K 2023-03-28 20:54:53,907 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 16a0b6726aa26da22f14d3e4178de945 in 448ms, sequenceid=375, compaction requested=true 2023-03-28 20:54:53,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:53,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:53,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:53,908 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:53,908 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:53,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:53,910 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64052 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:53,910 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:53,910 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:53,910 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:53,910 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,910 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,910 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33e397ba6b9f42dc8a87d3542af71334, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=62.6 K 2023-03-28 20:54:53,910 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/9f3552a5b46140b583a5b0ce8c28b3bb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=60.2 K 2023-03-28 20:54:53,911 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 33e397ba6b9f42dc8a87d3542af71334, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890909 2023-03-28 20:54:53,911 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 9f3552a5b46140b583a5b0ce8c28b3bb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890909 2023-03-28 20:54:53,911 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting aa5b82e5d19441c1b1af461909c40587, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680036890943 2023-03-28 20:54:53,911 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c3672cd8b8a5425c9e87a6cad62fec60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680036890943 2023-03-28 20:54:53,911 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ac1f8bc34b09416cab71a894422f67d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680036891090 2023-03-28 20:54:53,912 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 07e95d0576ab4fa19b084e14c154b89b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680036891090 2023-03-28 20:54:53,912 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c988989f510f4f78950bbd54fa37cb12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680036892248 2023-03-28 20:54:53,912 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ab2bc385243744eabc1f3e1e85b126b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680036892248 2023-03-28 20:54:53,912 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 56d8417312e9491a9534c4f8cbd41640, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893411 2023-03-28 20:54:53,912 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cb123333a6414403b924fff1caa1de2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893415 2023-03-28 20:54:53,921 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:53,922 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#291 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:53,953 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a94a28b4d0ee415eaf346b0559cd0315 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a94a28b4d0ee415eaf346b0559cd0315 2023-03-28 20:54:53,958 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into a94a28b4d0ee415eaf346b0559cd0315(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:53,958 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:53,958 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=11, startTime=1680036893908; duration=0sec 2023-03-28 20:54:53,958 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:53,958 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:53,960 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:53,960 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:53,960 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:53,960 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1ca9977218bb4168abd8fd4a65ce0c4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=60.2 K 2023-03-28 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1ca9977218bb4168abd8fd4a65ce0c4a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680036890909 2023-03-28 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 22d7dab36c6d497f9645183397d4882d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680036890943 2023-03-28 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4f6a31938780415d9a61a5d962855a12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680036891090 2023-03-28 20:54:53,962 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 3e6909ebc3b0460e87f8003fa9cf0276, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680036892248 2023-03-28 20:54:53,963 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a926e23988144743b476aa2d72e13dbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893415 2023-03-28 20:54:53,966 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:53,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:53,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:53,970 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:53,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:53,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:53,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:53,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,978 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:53,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:53,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:53,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:53,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:53,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:54,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:54,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:54,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:54,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:54,029 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,029 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:54,030 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:54,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:54,034 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/6aab56a649ad435cb6e9e8e190590273 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/6aab56a649ad435cb6e9e8e190590273 2023-03-28 20:54:54,045 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 6aab56a649ad435cb6e9e8e190590273(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:54,045 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:54,045 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=11, startTime=1680036893908; duration=0sec 2023-03-28 20:54:54,046 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:54,048 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/0fcd79eb0e51451891f56d63151397d5 2023-03-28 20:54:54,083 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e601a3da7ebb400f92ff0cefa112f98a 2023-03-28 20:54:54,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036954094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036894083, max timestamp=1680036894096Num uniques 859; 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036894083, max timestamp=1680036894096Num uniques 859; 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036894083, max timestamp=1680036894096Num uniques 859; 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:54,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:54:54,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036954115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036954116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036954117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680036954199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036954219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036954219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036954220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,332 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/5b2d57ef3d31467f8f55204e1ea332e9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/5b2d57ef3d31467f8f55204e1ea332e9 2023-03-28 20:54:54,336 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 5b2d57ef3d31467f8f55204e1ea332e9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:54,336 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:54,336 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=11, startTime=1680036893908; duration=0sec 2023-03-28 20:54:54,336 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:54,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036954401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036954422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036954423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036954424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,495 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/8a5d12bc508b40dda820b2b6a86a2fab 2023-03-28 20:54:54,499 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/0fcd79eb0e51451891f56d63151397d5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5 2023-03-28 20:54:54,503 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5, entries=150, sequenceid=387, filesize=11.9 K 2023-03-28 20:54:54,503 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e601a3da7ebb400f92ff0cefa112f98a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a 2023-03-28 20:54:54,507 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a, entries=150, sequenceid=387, filesize=11.9 K 2023-03-28 20:54:54,507 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/8a5d12bc508b40dda820b2b6a86a2fab as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab 2023-03-28 20:54:54,510 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab, entries=150, sequenceid=387, filesize=11.9 K 2023-03-28 20:54:54,511 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 16a0b6726aa26da22f14d3e4178de945 in 482ms, sequenceid=387, compaction requested=false 2023-03-28 20:54:54,511 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:54,511 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:54,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:54,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:54,511 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,514 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b348e22[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:54,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:54,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,517 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:54,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,518 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:54,521 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:54,521 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:54,521 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:54,521 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:54,522 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:54,523 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,523 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,523 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,524 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:54,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:54,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,529 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:54,529 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:54,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:54,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,629 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:54,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:54,634 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:54,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:54,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:54,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,634 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,635 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,636 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:54,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:54,645 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-28 20:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:54,646 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:54,646 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:54,646 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:54,656 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/53acc507110442749c0e67b1a6c22c6c 2023-03-28 20:54:54,693 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c075f91e959d4b09bf4bc4811d1aae14 2023-03-28 20:54:54,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:54,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:54,706 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/0c1fe4b8fe954011b6aa8bd6d01b315a 2023-03-28 20:54:54,711 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/53acc507110442749c0e67b1a6c22c6c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c 2023-03-28 20:54:54,717 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c, entries=150, sequenceid=424, filesize=11.9 K 2023-03-28 20:54:54,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c075f91e959d4b09bf4bc4811d1aae14 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14 2023-03-28 20:54:54,729 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14, entries=150, sequenceid=424, filesize=11.9 K 2023-03-28 20:54:54,730 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/0c1fe4b8fe954011b6aa8bd6d01b315a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a 2023-03-28 20:54:54,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,733 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a, entries=150, sequenceid=424, filesize=11.9 K 2023-03-28 20:54:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036954728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036954729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,734 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 16a0b6726aa26da22f14d3e4178de945 in 89ms, sequenceid=424, compaction requested=true 2023-03-28 20:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:54,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:54,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:54,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:54,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:54,740 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:54,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216f9742[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:54,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:54,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:54,749 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,749 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:54,751 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:54,756 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:54,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:54,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:54,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:54,756 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:54,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:54,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,765 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:54,765 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,765 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036954770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036954774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036954835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,865 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:54,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:54,869 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:54,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:54,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:54,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,870 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,873 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,873 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:54,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:54,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:54,874 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,874 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:54,874 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:54,874 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:54,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036954875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3523ff96[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:54,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:54,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:54,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036954878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:54,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:54,881 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:54,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:54,883 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:54,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:54,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:54,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:54,884 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:54,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:54,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,890 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:54,890 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:54,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:54,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:54,991 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:54,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:54,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:54,994 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:54,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:54,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:54,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:54,995 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,995 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,998 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:54,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:54,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:54,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:54,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,002 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc2e120[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,004 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,004 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,007 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,010 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,010 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,012 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,012 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,016 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,016 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036955037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036955078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036955081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,116 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,119 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,121 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,127 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63482f04[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,129 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,129 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,130 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,130 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,134 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,134 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,134 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,134 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,134 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,145 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,145 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/e250ab4c406340c5bdcda338cea9ca0d 2023-03-28 20:54:55,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f65d683affc140a8867f0a91e2321a3f 2023-03-28 20:54:55,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4c8abf8e9173494daa57ce5841bc7215 2023-03-28 20:54:55,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/e250ab4c406340c5bdcda338cea9ca0d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d 2023-03-28 20:54:55,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d, entries=200, sequenceid=436, filesize=14.3 K 2023-03-28 20:54:55,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f65d683affc140a8867f0a91e2321a3f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f 2023-03-28 20:54:55,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:54:55,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/4c8abf8e9173494daa57ce5841bc7215 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215 2023-03-28 20:54:55,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:54:55,203 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 468ms, sequenceid=436, compaction requested=true 2023-03-28 20:54:55,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:55,203 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:55,203 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:55,203 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:55,203 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:55,203 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:55,204 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:55,205 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:55,205 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,205 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/5b2d57ef3d31467f8f55204e1ea332e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=48.4 K 2023-03-28 20:54:55,205 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:55,206 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:55,206 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,206 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a94a28b4d0ee415eaf346b0559cd0315, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=50.8 K 2023-03-28 20:54:55,206 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5b2d57ef3d31467f8f55204e1ea332e9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893415 2023-03-28 20:54:55,206 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a94a28b4d0ee415eaf346b0559cd0315, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893415 2023-03-28 20:54:55,207 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e601a3da7ebb400f92ff0cefa112f98a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680036893471 2023-03-28 20:54:55,207 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0fcd79eb0e51451891f56d63151397d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680036893471 2023-03-28 20:54:55,207 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c075f91e959d4b09bf4bc4811d1aae14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680036894104 2023-03-28 20:54:55,209 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 53acc507110442749c0e67b1a6c22c6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680036894104 2023-03-28 20:54:55,209 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f65d683affc140a8867f0a91e2321a3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036894727 2023-03-28 20:54:55,209 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e250ab4c406340c5bdcda338cea9ca0d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036894727 2023-03-28 20:54:55,222 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#302 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:55,222 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#303 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:55,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:54:55,245 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,264 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,264 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,266 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,266 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036955272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1cae8b92413d4de79df470cd847af57b 2023-03-28 20:54:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec071a8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,302 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,302 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:55,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,310 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/59b25545e1794a11961287b7331084d1 2023-03-28 20:54:55,310 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/e7289172e16047ba90995f6ac77257d3 2023-03-28 20:54:55,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1cae8b92413d4de79df470cd847af57b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b 2023-03-28 20:54:55,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b, entries=150, sequenceid=457, filesize=11.9 K 2023-03-28 20:54:55,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/59b25545e1794a11961287b7331084d1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1 2023-03-28 20:54:55,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1, entries=150, sequenceid=457, filesize=11.9 K 2023-03-28 20:54:55,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/e7289172e16047ba90995f6ac77257d3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3 2023-03-28 20:54:55,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3, entries=150, sequenceid=457, filesize=11.9 K 2023-03-28 20:54:55,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 16a0b6726aa26da22f14d3e4178de945 in 97ms, sequenceid=457, compaction requested=true 2023-03-28 20:54:55,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:55,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-28 20:54:55,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-28 20:54:55,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-28 20:54:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:55,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:55,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:55,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/262ef59ea6bb4851af8f4f7f3ceb24da 2023-03-28 20:54:55,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036955389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036955389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036955390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036955393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,410 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,413 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,422 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b5e6a9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,425 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,429 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,429 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,435 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,435 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,435 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036955494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036955496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036955496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036955498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,536 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,539 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,545 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,546 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc5defe[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,555 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,556 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:55,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,562 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,562 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,562 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,562 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,637 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/83c09c9032054408b3750c61f0dba7f7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/83c09c9032054408b3750c61f0dba7f7 2023-03-28 20:54:55,638 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/f8cea17974a0478bb3ac75304b88e33c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f8cea17974a0478bb3ac75304b88e33c 2023-03-28 20:54:55,643 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 83c09c9032054408b3750c61f0dba7f7(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:55,643 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into f8cea17974a0478bb3ac75304b88e33c(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:55,643 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:55,643 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=12, startTime=1680036895203; duration=0sec 2023-03-28 20:54:55,643 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-28 20:54:55,643 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:54:55,643 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:55,643 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=12, startTime=1680036895203; duration=0sec 2023-03-28 20:54:55,644 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-03-28 20:54:55,645 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. because compaction request was cancelled 2023-03-28 20:54:55,645 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/6aab56a649ad435cb6e9e8e190590273, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=60.3 K 2023-03-28 20:54:55,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 6aab56a649ad435cb6e9e8e190590273, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680036893415 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 8a5d12bc508b40dda820b2b6a86a2fab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680036893471 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. because compaction request was cancelled 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-28 20:54:55,646 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0c1fe4b8fe954011b6aa8bd6d01b315a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680036894104 2023-03-28 20:54:55,647 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4c8abf8e9173494daa57ce5841bc7215, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036894727 2023-03-28 20:54:55,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:54:55,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:54:55,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. because compaction request was cancelled 2023-03-28 20:54:55,647 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e7289172e16047ba90995f6ac77257d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036894763 2023-03-28 20:54:55,654 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#309 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:55,662 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/63a68a697e8e40febfc34d6c9e2016fb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/63a68a697e8e40febfc34d6c9e2016fb 2023-03-28 20:54:55,662 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,665 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,667 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 63a68a697e8e40febfc34d6c9e2016fb(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:55,667 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:55,667 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=11, startTime=1680036895339; duration=0sec 2023-03-28 20:54:55,667 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:55,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,669 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,672 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd7e40d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,676 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,678 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,680 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,681 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,687 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,687 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,687 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,688 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,692 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,692 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036955697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036955700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:55,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036955701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036955702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a77b7d948a7b40b7a22862eea951623b 2023-03-28 20:54:55,793 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,795 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,798 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,799 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,799 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,799 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,799 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,800 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7992e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,805 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,809 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,811 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,812 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,817 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,818 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,818 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:55,918 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:55,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:55,921 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:55,921 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,921 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:55,921 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:55,921 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:55,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:55,926 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,926 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:55,926 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:55,926 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:55,928 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1cb13d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:55,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:55,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,931 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:55,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:55,935 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:55,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:55,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:55,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:55,935 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:55,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:55,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:55,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:55,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:55,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:55,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:55,944 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:55,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:55,945 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:56,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036956001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036956004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036956006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036956007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,045 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:56,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:56,048 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:56,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:56,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:56,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,048 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,049 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,050 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:56,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:56,051 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,051 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:56,052 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:56,052 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,053 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2fac8e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:56,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:56,055 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,058 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:56,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:56,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:56,062 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:56,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:56,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:56,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:56,062 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:56,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:56,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:56,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,068 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,068 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,169 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:56,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:56,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:56,172 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:56,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/e1d47af6cd45428f9b47d9a1b48958fc 2023-03-28 20:54:56,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:56,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:56,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/262ef59ea6bb4851af8f4f7f3ceb24da as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:56,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ad64a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:56,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:56,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da, entries=200, sequenceid=469, filesize=14.3 K 2023-03-28 20:54:56,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:56,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:56,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a77b7d948a7b40b7a22862eea951623b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b 2023-03-28 20:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:56,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b, entries=150, sequenceid=469, filesize=11.9 K 2023-03-28 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:56,186 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:56,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:56,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:56,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:56,186 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:56,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/e1d47af6cd45428f9b47d9a1b48958fc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc 2023-03-28 20:54:56,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc, entries=150, sequenceid=469, filesize=11.9 K 2023-03-28 20:54:56,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16a0b6726aa26da22f14d3e4178de945 in 845ms, sequenceid=469, compaction requested=true 2023-03-28 20:54:56,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:56,190 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:56,190 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:56,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,191 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,191 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,191 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,191 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:56,191 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,191 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,192 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,191 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,192 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,192 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f8cea17974a0478bb3ac75304b88e33c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=39.0 K 2023-03-28 20:54:56,191 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,192 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/83c09c9032054408b3750c61f0dba7f7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.7 K 2023-03-28 20:54:56,192 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f8cea17974a0478bb3ac75304b88e33c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036894727 2023-03-28 20:54:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,192 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 83c09c9032054408b3750c61f0dba7f7, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036894727 2023-03-28 20:54:56,193 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1cae8b92413d4de79df470cd847af57b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036894763 2023-03-28 20:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,193 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 59b25545e1794a11961287b7331084d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036894763 2023-03-28 20:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,193 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 262ef59ea6bb4851af8f4f7f3ceb24da, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680036895262 2023-03-28 20:54:56,193 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a77b7d948a7b40b7a22862eea951623b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680036895264 2023-03-28 20:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:56,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:56,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,197 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:56,197 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:56,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,200 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#311 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:56,202 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:56,213 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/6e9385b71d0a452b892cde4bf42d52f9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/6e9385b71d0a452b892cde4bf42d52f9 2023-03-28 20:54:56,213 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/4c9f9aea92434599b884f73a413df07c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/4c9f9aea92434599b884f73a413df07c 2023-03-28 20:54:56,217 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 4c9f9aea92434599b884f73a413df07c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:56,217 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,217 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036896190; duration=0sec 2023-03-28 20:54:56,218 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 6e9385b71d0a452b892cde4bf42d52f9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-28 20:54:56,218 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036896190; duration=0sec 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:54:56,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. because compaction request was cancelled 2023-03-28 20:54:56,297 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:56,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:56,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:56,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:56,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:56,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:56,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,303 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:56,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:56,306 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,314 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a3fd3c456571467189226c886eaf8686 2023-03-28 20:54:56,325 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/fb8096f9b47e4f449881e3faa298a336 2023-03-28 20:54:56,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:56,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:56,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036956513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036956515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036956515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036956516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036956617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036956619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036956619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036956619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,737 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/efbb916d71604942a2191592b65af107 2023-03-28 20:54:56,741 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/a3fd3c456571467189226c886eaf8686 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686 2023-03-28 20:54:56,744 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686, entries=150, sequenceid=494, filesize=11.9 K 2023-03-28 20:54:56,745 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/fb8096f9b47e4f449881e3faa298a336 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336 2023-03-28 20:54:56,748 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336, entries=150, sequenceid=494, filesize=11.9 K 2023-03-28 20:54:56,749 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/efbb916d71604942a2191592b65af107 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107 2023-03-28 20:54:56,752 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107, entries=150, sequenceid=494, filesize=11.9 K 2023-03-28 20:54:56,752 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 16a0b6726aa26da22f14d3e4178de945 in 446ms, sequenceid=494, compaction requested=true 2023-03-28 20:54:56,753 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,753 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:56,755 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34084680[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:56,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:56,757 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:56,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:56,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:56,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:56,760 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:56,763 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:56,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:56,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:56,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:56,763 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:56,764 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,764 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,764 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:56,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,770 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:56,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,770 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:56,770 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,770 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:56,823 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:54:56,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:56,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:56,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:56,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/af7153d50fe34808845c958db3f19b03 2023-03-28 20:54:56,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036956844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036956845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036956846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036956846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/feeabe1e2d2e458cb44e5bdd27bce213 2023-03-28 20:54:56,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ccde2cdfa2d64591bd9333e5072fddea 2023-03-28 20:54:56,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/af7153d50fe34808845c958db3f19b03 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03 2023-03-28 20:54:56,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03, entries=200, sequenceid=509, filesize=14.3 K 2023-03-28 20:54:56,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/feeabe1e2d2e458cb44e5bdd27bce213 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213 2023-03-28 20:54:56,870 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:56,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:56,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:56,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:56,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:56,876 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213, entries=150, sequenceid=509, filesize=11.9 K 2023-03-28 20:54:56,878 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ccde2cdfa2d64591bd9333e5072fddea as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:56,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea, entries=150, sequenceid=509, filesize=11.9 K 2023-03-28 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 16a0b6726aa26da22f14d3e4178de945 in 60ms, sequenceid=509, compaction requested=true 2023-03-28 20:54:56,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:56,883 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:56,883 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:56,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:56,884 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:56,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:56,885 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a739c65[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:56,885 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:56,885 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/6e9385b71d0a452b892cde4bf42d52f9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.8 K 2023-03-28 20:54:56,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:56,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,885 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,885 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/4c9f9aea92434599b884f73a413df07c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=39.1 K 2023-03-28 20:54:56,885 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6e9385b71d0a452b892cde4bf42d52f9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680036895264 2023-03-28 20:54:56,885 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4c9f9aea92434599b884f73a413df07c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680036895264 2023-03-28 20:54:56,886 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fb8096f9b47e4f449881e3faa298a336, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680036895388 2023-03-28 20:54:56,886 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a3fd3c456571467189226c886eaf8686, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680036895388 2023-03-28 20:54:56,886 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting feeabe1e2d2e458cb44e5bdd27bce213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:56,886 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting af7153d50fe34808845c958db3f19b03, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:56,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:56,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:56,889 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,893 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#319 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:56,897 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:56,900 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:56,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:56,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:56,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:56,900 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:56,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,908 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:56,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:56,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:56,908 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:56,909 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:56,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:56,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:56,910 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/7855c0e288d44c5786784667027f1f4c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7855c0e288d44c5786784667027f1f4c 2023-03-28 20:54:56,914 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 7855c0e288d44c5786784667027f1f4c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:56,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,914 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036896883; duration=0sec 2023-03-28 20:54:56,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:56,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:54:56,915 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:54:56,915 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:56,915 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:56,915 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/63a68a697e8e40febfc34d6c9e2016fb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=48.6 K 2023-03-28 20:54:56,916 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 63a68a697e8e40febfc34d6c9e2016fb, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036894763 2023-03-28 20:54:56,916 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e1d47af6cd45428f9b47d9a1b48958fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1680036895264 2023-03-28 20:54:56,916 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting efbb916d71604942a2191592b65af107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680036895388 2023-03-28 20:54:56,917 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ccde2cdfa2d64591bd9333e5072fddea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:56,924 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:56,942 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a3fff8b138dd4cdc90e719dac4509a66 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3fff8b138dd4cdc90e719dac4509a66 2023-03-28 20:54:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:56,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:56,959 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into a3fff8b138dd4cdc90e719dac4509a66(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:56,959 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:56,959 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=12, startTime=1680036896884; duration=0sec 2023-03-28 20:54:56,959 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:56,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036956968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/54cc3e141d614b72bc69297a5f5879cc 2023-03-28 20:54:56,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036956969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036956970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036956970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:56,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c9702f027fcd4ce79bd306636029776e 2023-03-28 20:54:57,009 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,024 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/de2925cb3fe84b55972c1c761000fad4 2023-03-28 20:54:57,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/54cc3e141d614b72bc69297a5f5879cc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc 2023-03-28 20:54:57,048 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc, entries=200, sequenceid=529, filesize=14.3 K 2023-03-28 20:54:57,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c9702f027fcd4ce79bd306636029776e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e 2023-03-28 20:54:57,052 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4f0079[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e, entries=150, sequenceid=529, filesize=11.9 K 2023-03-28 20:54:57,055 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,056 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,056 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/de2925cb3fe84b55972c1c761000fad4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4 2023-03-28 20:54:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,068 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,068 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4, entries=150, sequenceid=529, filesize=11.9 K 2023-03-28 20:54:57,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 16a0b6726aa26da22f14d3e4178de945 in 119ms, sequenceid=529, compaction requested=false 2023-03-28 20:54:57,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:57,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,070 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:57,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:57,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:57,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,078 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,079 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,079 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036957094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036957094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036957097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036957098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,179 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,182 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,184 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,185 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,185 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,185 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,185 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,187 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43795b1c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,191 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,193 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,195 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,196 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036957199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036957199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036957201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036957201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,204 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,204 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,204 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,304 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/c22538478c8a431596887946bacdf93f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c22538478c8a431596887946bacdf93f 2023-03-28 20:54:57,305 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,309 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,310 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into c22538478c8a431596887946bacdf93f(size=13.1 K), total size for store is 27.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:57,310 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:57,310 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036896883; duration=0sec 2023-03-28 20:54:57,310 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:57,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,313 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,313 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,313 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,313 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,316 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b339b5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,321 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,324 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,324 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,326 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,326 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,326 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,329 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,334 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,334 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,334 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036957402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036957402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036957402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036957405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,434 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,437 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,441 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e51bd2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,450 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,453 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,453 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,455 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,455 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,455 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,455 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,459 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,459 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,460 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,459 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680036957486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,488 DEBUG [Thread-1721] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:54:57,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/66dc1774d57a40e4b3f132baa2298ec3 2023-03-28 20:54:57,560 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,563 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,567 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,569 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,570 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fc811f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,579 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,580 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,580 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,582 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,586 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,586 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,586 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,686 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,689 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036957704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036957704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036957705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:57,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036957709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f548f67[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,719 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,722 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,722 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,725 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,726 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,729 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,729 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,729 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,830 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,844 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,847 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,847 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:57,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:57,857 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:57,857 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:57,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38675a9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:57,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:57,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,864 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:57,868 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:57,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:57,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:57,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:57,869 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:57,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:57,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:57,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:57,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:57,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:57,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:57,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:57,881 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:57,881 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:57,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:57,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/1c1f552d80ec4878a6b2d6c75d03aeef 2023-03-28 20:54:57,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/1451b1a835414ec0addff636222b18b6 2023-03-28 20:54:57,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/66dc1774d57a40e4b3f132baa2298ec3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3 2023-03-28 20:54:57,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3, entries=150, sequenceid=545, filesize=11.9 K 2023-03-28 20:54:57,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/1c1f552d80ec4878a6b2d6c75d03aeef as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef 2023-03-28 20:54:57,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef, entries=150, sequenceid=545, filesize=11.9 K 2023-03-28 20:54:57,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/1451b1a835414ec0addff636222b18b6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6 2023-03-28 20:54:57,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6, entries=150, sequenceid=545, filesize=11.9 K 2023-03-28 20:54:57,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 16a0b6726aa26da22f14d3e4178de945 in 876ms, sequenceid=545, compaction requested=true 2023-03-28 20:54:57,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:57,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:57,951 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:57,951 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:57,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:57,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:57,952 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:57,952 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:57,952 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,952 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c22538478c8a431596887946bacdf93f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=39.2 K 2023-03-28 20:54:57,953 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:57,953 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:57,953 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:57,953 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7855c0e288d44c5786784667027f1f4c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.9 K 2023-03-28 20:54:57,953 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c22538478c8a431596887946bacdf93f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:57,953 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7855c0e288d44c5786784667027f1f4c, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:57,954 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 54cc3e141d614b72bc69297a5f5879cc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680036896843 2023-03-28 20:54:57,954 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c9702f027fcd4ce79bd306636029776e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680036896844 2023-03-28 20:54:57,954 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 66dc1774d57a40e4b3f132baa2298ec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:57,954 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1c1f552d80ec4878a6b2d6c75d03aeef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:57,963 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#328 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:57,965 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#329 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:57,981 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:57,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:57,996 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,997 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,998 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:57,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:57,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:57,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:57,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:58,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:58,020 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,020 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:58,021 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,037 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/f903d8e7bd774600a62066d791fc4d0a 2023-03-28 20:54:58,057 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/75e7d84cafab4435baa5ca047d54d791 2023-03-28 20:54:58,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:58,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036958220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036958220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036958221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036958222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036958323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036958323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036958325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036958325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,410 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/a3f8c253ddb34290aae015e49e534b2c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a3f8c253ddb34290aae015e49e534b2c 2023-03-28 20:54:58,415 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into a3f8c253ddb34290aae015e49e534b2c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:58,415 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:58,415 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036897951; duration=0sec 2023-03-28 20:54:58,415 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:58,415 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:58,418 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:58,418 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:58,418 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,418 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3fff8b138dd4cdc90e719dac4509a66, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.8 K 2023-03-28 20:54:58,418 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1bb1a7b30e28442b9e045e4ef3594a5e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bb1a7b30e28442b9e045e4ef3594a5e 2023-03-28 20:54:58,420 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a3fff8b138dd4cdc90e719dac4509a66, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1680036896507 2023-03-28 20:54:58,420 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting de2925cb3fe84b55972c1c761000fad4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680036896844 2023-03-28 20:54:58,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1451b1a835414ec0addff636222b18b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:58,424 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 1bb1a7b30e28442b9e045e4ef3594a5e(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:58,424 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:58,424 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036897951; duration=0sec 2023-03-28 20:54:58,425 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:58,426 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:58,442 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/ff2d29d8bbd34308a74da478e5fa969f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ff2d29d8bbd34308a74da478e5fa969f 2023-03-28 20:54:58,446 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into ff2d29d8bbd34308a74da478e5fa969f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:58,446 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:58,446 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=13, startTime=1680036897951; duration=0sec 2023-03-28 20:54:58,446 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:58,468 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3935402521604a249c3fccc50230da2e 2023-03-28 20:54:58,472 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/f903d8e7bd774600a62066d791fc4d0a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a 2023-03-28 20:54:58,475 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a, entries=150, sequenceid=563, filesize=11.9 K 2023-03-28 20:54:58,476 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/75e7d84cafab4435baa5ca047d54d791 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791 2023-03-28 20:54:58,480 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791, entries=150, sequenceid=563, filesize=11.9 K 2023-03-28 20:54:58,481 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3935402521604a249c3fccc50230da2e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e 2023-03-28 20:54:58,484 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e, entries=150, sequenceid=563, filesize=11.9 K 2023-03-28 20:54:58,485 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 16a0b6726aa26da22f14d3e4178de945 in 465ms, sequenceid=563, compaction requested=false 2023-03-28 20:54:58,485 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:58,485 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:58,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,493 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:58,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5866c580[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:58,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:58,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:58,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:58,498 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:58,500 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:58,500 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:58,500 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:58,500 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:58,500 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:58,502 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,502 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,502 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,507 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:58,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:58,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:58,508 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:58,508 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:58,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:58,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:58,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/df23346cc70841dfa9ffb0f411aef56e 2023-03-28 20:54:58,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036958542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036958542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036958545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036958545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,608 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:58,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:58,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:58,611 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:58,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:58,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:58,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,613 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:58,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:58,615 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,615 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:58,615 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:58,615 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:58,616 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad053ea[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:58,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:58,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:58,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:58,622 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:58,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:58,624 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:58,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:58,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:58,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:58,624 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:58,626 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,626 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,626 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,626 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:58,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,630 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,630 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,630 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:58,630 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:58,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036958646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036958646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036958648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036958648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,731 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:58,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:58,734 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:58,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:58,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:58,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,736 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,736 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:58,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:58,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:58,737 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,737 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:58,738 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:58,738 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f4d860[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:58,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:58,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:58,747 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,747 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:58,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:58,750 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:58,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:58,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:58,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:58,751 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:58,752 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,752 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,752 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,753 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,754 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,758 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:58,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,759 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:58,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036958850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036958850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036958851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036958852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,859 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:58,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:58,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:58,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:58,862 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:58,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:58,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:58,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,864 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,864 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,864 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,864 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:58,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:58,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:58,867 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21efdcca[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:58,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:58,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,870 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:58,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:58,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:58,870 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:58,875 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:58,878 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:58,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:58,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:58,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:58,878 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:58,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:58,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:58,884 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:58,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,885 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:58,885 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:58,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/0acf20e55ebd47e59e531a33fbd75b44 2023-03-28 20:54:58,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70 2023-03-28 20:54:58,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/df23346cc70841dfa9ffb0f411aef56e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e 2023-03-28 20:54:58,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e, entries=250, sequenceid=583, filesize=16.7 K 2023-03-28 20:54:58,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/0acf20e55ebd47e59e531a33fbd75b44 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44 2023-03-28 20:54:58,985 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:58,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:58,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:58,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44, entries=150, sequenceid=583, filesize=11.9 K 2023-03-28 20:54:58,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:58,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:58,988 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:58,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:58,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:58,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70 2023-03-28 20:54:58,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:58,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:58,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:58,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:54:58,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70, entries=150, sequenceid=583, filesize=11.9 K 2023-03-28 20:54:58,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 16a0b6726aa26da22f14d3e4178de945 in 465ms, sequenceid=583, compaction requested=true 2023-03-28 20:54:58,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:58,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:58,995 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:58,995 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:58,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:58,995 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:58,997 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42732 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:58,997 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:58,997 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:54:58,997 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:54:58,997 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:58,997 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bb1a7b30e28442b9e045e4ef3594a5e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=41.7 K 2023-03-28 20:54:58,997 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:58,997 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a3f8c253ddb34290aae015e49e534b2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=37.0 K 2023-03-28 20:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:58,998 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1bb1a7b30e28442b9e045e4ef3594a5e, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4b47b2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:54:58,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:54:58,998 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a3f8c253ddb34290aae015e49e534b2c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:58,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:58,998 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f903d8e7bd774600a62066d791fc4d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1680036897083 2023-03-28 20:54:58,999 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting df23346cc70841dfa9ffb0f411aef56e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:54:58,999 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 75e7d84cafab4435baa5ca047d54d791, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1680036897083 2023-03-28 20:54:59,000 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:59,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:54:59,000 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0acf20e55ebd47e59e531a33fbd75b44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:54:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:54:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:54:59,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:54:59,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:54:59,006 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:54:59,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:54:59,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:54:59,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:54:59,006 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:54:59,009 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#337 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:59,010 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,010 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,010 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:59,011 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#338 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:54:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:59,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:54:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:54:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:54:59,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:59,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:54:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:59,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:54:59,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:59,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:59,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:54:59,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,020 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:54:59,020 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:54:59,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:54:59,120 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:54:59,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:54:59,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:54:59,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:54:59,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:54:59,124 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:54:59,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:59,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:54:59,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:54:59,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:59,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,131 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:54:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:54:59,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:59,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:54:59,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:54:59,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:54:59,134 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:59,134 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:54:59,134 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:54:59,146 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/fc938e629c6c47488f8d1be1d9a7f5e5 2023-03-28 20:54:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:54:59,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036959166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036959167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036959169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036959170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036959270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036959271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036959272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036959273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,431 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/38dcb4f2aabb452c804b8a759890380d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/38dcb4f2aabb452c804b8a759890380d 2023-03-28 20:54:59,431 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/0a7e2e7595af42eb81cdbc3b1353655c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0a7e2e7595af42eb81cdbc3b1353655c 2023-03-28 20:54:59,435 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 38dcb4f2aabb452c804b8a759890380d(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:59,435 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:59,435 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036898995; duration=0sec 2023-03-28 20:54:59,435 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 0a7e2e7595af42eb81cdbc3b1353655c(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:59,435 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:54:59,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:59,435 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036898995; duration=0sec 2023-03-28 20:54:59,435 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:54:59,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:59,436 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:54:59,436 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:54:59,436 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:59,436 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ff2d29d8bbd34308a74da478e5fa969f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=36.9 K 2023-03-28 20:54:59,437 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ff2d29d8bbd34308a74da478e5fa969f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680036896955 2023-03-28 20:54:59,437 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3935402521604a249c3fccc50230da2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1680036897083 2023-03-28 20:54:59,437 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a0bae98a4cbc4ce9a1cf18ffb6a04c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:54:59,442 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:54:59,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036959474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036959474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036959474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036959477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,563 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e0bd61f144f848cbbe188d8e793666eb 2023-03-28 20:54:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036959777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036959778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036959778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:54:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036959781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,851 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/9b8c2de408b84094830986dfa1f094f2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/9b8c2de408b84094830986dfa1f094f2 2023-03-28 20:54:59,855 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into 9b8c2de408b84094830986dfa1f094f2(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:54:59,855 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:59,855 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=13, startTime=1680036898995; duration=0sec 2023-03-28 20:54:59,856 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:54:59,975 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/614a5a89275c400e83555e48f7841c38 2023-03-28 20:54:59,979 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/fc938e629c6c47488f8d1be1d9a7f5e5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5 2023-03-28 20:54:59,982 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5, entries=150, sequenceid=598, filesize=11.9 K 2023-03-28 20:54:59,983 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e0bd61f144f848cbbe188d8e793666eb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb 2023-03-28 20:54:59,989 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb, entries=150, sequenceid=598, filesize=11.9 K 2023-03-28 20:54:59,990 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/614a5a89275c400e83555e48f7841c38 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38 2023-03-28 20:54:59,998 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38, entries=150, sequenceid=598, filesize=11.9 K 2023-03-28 20:54:59,999 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 16a0b6726aa26da22f14d3e4178de945 in 865ms, sequenceid=598, compaction requested=false 2023-03-28 20:54:59,999 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:54:59,999 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:54:59,999 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:54:59,999 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:54:59,999 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:54:59,999 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:54:59,999 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,001 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@571c02b2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,003 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,003 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,004 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,004 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,004 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,006 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,008 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,008 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,008 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,016 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,016 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,116 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,119 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,121 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,123 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:00,123 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,133 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/eebd5411982749d1ab55f783ff76f207 2023-03-28 20:55:00,150 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c1fce4f8b2744fcd853313f7316ae4c6 2023-03-28 20:55:00,159 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3ffb609a8cd64049ba51a587e7a2a267 2023-03-28 20:55:00,162 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/eebd5411982749d1ab55f783ff76f207 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207 2023-03-28 20:55:00,166 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207, entries=150, sequenceid=618, filesize=11.9 K 2023-03-28 20:55:00,166 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c1fce4f8b2744fcd853313f7316ae4c6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6 2023-03-28 20:55:00,170 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6, entries=150, sequenceid=618, filesize=11.9 K 2023-03-28 20:55:00,171 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3ffb609a8cd64049ba51a587e7a2a267 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267 2023-03-28 20:55:00,178 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267, entries=150, sequenceid=618, filesize=11.9 K 2023-03-28 20:55:00,179 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 16a0b6726aa26da22f14d3e4178de945 in 56ms, sequenceid=618, compaction requested=true 2023-03-28 20:55:00,179 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:00,179 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,179 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601ce7e9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,194 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,197 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,197 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,203 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,207 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,207 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,207 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:00,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:00,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ec26fe0428cf4d68bcea5f46bacea8a0 2023-03-28 20:55:00,308 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,311 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036960312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036960313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036900300, max timestamp=1680036900315Num uniques 859; 2023-03-28 20:55:00,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:00,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036900300, max timestamp=1680036900315Num uniques 859; 2023-03-28 20:55:00,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036900300, max timestamp=1680036900315Num uniques 859; 2023-03-28 20:55:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:00,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fca6eac[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,322 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,323 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,326 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,328 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,330 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,332 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,332 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,332 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,332 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,333 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,333 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036960360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036960361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036960417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036960417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,433 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,436 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,440 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,440 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,440 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,440 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,442 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa9d830[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,448 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,450 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,450 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,451 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,451 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,452 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,452 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,452 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,457 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,457 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,457 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036960464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036960466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,558 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,562 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,562 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,562 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,562 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,563 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,564 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d175645[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,571 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,571 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,571 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,575 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,575 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,579 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,584 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,585 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,585 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,585 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036960620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036960621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036960667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036960671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,686 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,689 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,692 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,692 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,692 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,692 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,694 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab3d9a1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,702 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,703 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,703 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,703 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,703 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,704 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,704 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,704 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,705 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,708 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/03548d4d7d234358b3e67579deeab118 2023-03-28 20:55:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,709 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,709 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,809 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,812 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,817 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,819 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e07b55[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,824 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,826 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,826 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,832 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,833 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,833 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036960923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036960926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,933 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:00,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:00,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:00,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:00,937 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:00,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:00,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:00,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:00,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:00,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:00,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:00,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:00,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:00,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:00,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@126c45e1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:00,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:00,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,946 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:00,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:00,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,948 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:00,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:00,950 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:00,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:00,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:00,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:00,950 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:00,951 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,951 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,952 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:00,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,953 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:00,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:00,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:00,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:00,962 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:00,962 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:00,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:00,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:00,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036960972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:00,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:00,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036960975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,062 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:01,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:01,065 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:01,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:01,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:01,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,067 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,067 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,067 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:01,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:01,069 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,069 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:01,069 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:01,069 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:01,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:01,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,070 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356e8c97[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:01,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:01,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:01,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:01,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:01,076 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:01,079 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:01,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:01,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:01,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:01,079 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:01,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,081 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:01,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,085 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:01,086 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:01,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/80d968de18304b918687baa45fc1d8c1 2023-03-28 20:55:01,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ec26fe0428cf4d68bcea5f46bacea8a0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0 2023-03-28 20:55:01,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0, entries=150, sequenceid=629, filesize=11.9 K 2023-03-28 20:55:01,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/03548d4d7d234358b3e67579deeab118 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118 2023-03-28 20:55:01,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118, entries=150, sequenceid=629, filesize=11.9 K 2023-03-28 20:55:01,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/80d968de18304b918687baa45fc1d8c1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1 2023-03-28 20:55:01,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1, entries=150, sequenceid=629, filesize=11.9 K 2023-03-28 20:55:01,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 16a0b6726aa26da22f14d3e4178de945 in 844ms, sequenceid=629, compaction requested=true 2023-03-28 20:55:01,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:01,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:01,132 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:01,132 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:01,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:01,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:01,133 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:01,133 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:55:01,134 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,134 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,134 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0a7e2e7595af42eb81cdbc3b1353655c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=49.0 K 2023-03-28 20:55:01,134 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/38dcb4f2aabb452c804b8a759890380d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=49.0 K 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0a7e2e7595af42eb81cdbc3b1353655c, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 38dcb4f2aabb452c804b8a759890380d, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fc938e629c6c47488f8d1be1d9a7f5e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1680036898541 2023-03-28 20:55:01,134 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e0bd61f144f848cbbe188d8e793666eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1680036898541 2023-03-28 20:55:01,135 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting eebd5411982749d1ab55f783ff76f207, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680036899161 2023-03-28 20:55:01,135 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c1fce4f8b2744fcd853313f7316ae4c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680036899161 2023-03-28 20:55:01,135 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ec26fe0428cf4d68bcea5f46bacea8a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:01,135 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 03548d4d7d234358b3e67579deeab118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:01,142 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:01,142 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#349 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:01,150 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/19d2949fea734918971f120d0efbb5c6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/19d2949fea734918971f120d0efbb5c6 2023-03-28 20:55:01,154 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into 19d2949fea734918971f120d0efbb5c6(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:01,154 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:01,154 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=12, startTime=1680036901132; duration=0sec 2023-03-28 20:55:01,154 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:01,154 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:01,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:01,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:55:01,155 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,155 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/9b8c2de408b84094830986dfa1f094f2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=48.9 K 2023-03-28 20:55:01,156 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 9b8c2de408b84094830986dfa1f094f2, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1680036898215 2023-03-28 20:55:01,156 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 614a5a89275c400e83555e48f7841c38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1680036898541 2023-03-28 20:55:01,157 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3ffb609a8cd64049ba51a587e7a2a267, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680036899161 2023-03-28 20:55:01,157 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 80d968de18304b918687baa45fc1d8c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:01,166 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#351 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:01,186 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:01,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:01,188 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:01,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:01,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:01,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,189 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,191 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,191 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:01,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:01,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:01,193 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,193 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:01,193 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:01,194 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:01,200 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ded0b39fc19d48b981fb66207ce6be2c 2023-03-28 20:55:01,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:01,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:01,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036961441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036961442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036961479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036961481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036961543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036961546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,550 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/46ea418b76c44ca48ab7a6dd7c0af736 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/46ea418b76c44ca48ab7a6dd7c0af736 2023-03-28 20:55:01,554 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into 46ea418b76c44ca48ab7a6dd7c0af736(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:01,554 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:01,555 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=12, startTime=1680036901132; duration=0sec 2023-03-28 20:55:01,555 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:01,574 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/eec861018819451c89f5e8ec485bc6e8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eec861018819451c89f5e8ec485bc6e8 2023-03-28 20:55:01,577 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into eec861018819451c89f5e8ec485bc6e8(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:01,578 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:01,578 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=12, startTime=1680036901132; duration=0sec 2023-03-28 20:55:01,578 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:01,614 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/b337b52c1cba46fab288dddc4561c4f8 2023-03-28 20:55:01,628 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3a3e3df84f434004ab6506470fc440a5 2023-03-28 20:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ded0b39fc19d48b981fb66207ce6be2c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c 2023-03-28 20:55:01,637 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c, entries=150, sequenceid=665, filesize=11.9 K 2023-03-28 20:55:01,638 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/b337b52c1cba46fab288dddc4561c4f8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8 2023-03-28 20:55:01,642 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8, entries=150, sequenceid=665, filesize=11.9 K 2023-03-28 20:55:01,642 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/3a3e3df84f434004ab6506470fc440a5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5 2023-03-28 20:55:01,647 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5, entries=150, sequenceid=665, filesize=11.9 K 2023-03-28 20:55:01,651 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 16a0b6726aa26da22f14d3e4178de945 in 458ms, sequenceid=665, compaction requested=false 2023-03-28 20:55:01,651 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:01,651 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:01,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:01,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:01,651 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:01,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3114254a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:01,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:01,655 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,656 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:01,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,656 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:01,658 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:01,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:01,662 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:01,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:01,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:01,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:01,662 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:01,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,671 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:01,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,672 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:01,672 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:01,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:01,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:01,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:01,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/2332628e33f941f6a1f81828c68fbd74 2023-03-28 20:55:01,772 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:01,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:01,776 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:01,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:01,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:01,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,778 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:01,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:01,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:01,779 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,779 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:01,779 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:01,779 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:01,781 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445bcaee[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:01,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:01,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,785 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,785 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:01,785 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:01,785 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:01,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,786 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:01,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:01,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:01,789 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:01,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:01,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:01,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:01,789 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:01,791 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,791 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,791 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,793 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036961790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036961792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:01,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,795 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,795 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:01,896 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:01,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036961895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036961895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:01,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:01,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:01,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:01,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,903 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:01,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:01,904 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,904 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. started... 2023-03-28 20:55:01,904 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. as already flushing 2023-03-28 20:55:01,904 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:01,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,905 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:01,905 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:01,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:01,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e8ccc9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:01,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:01,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:01,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:01,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:01,912 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:01,914 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:01,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:01,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:01,916 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:01,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:01,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:01,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:01,917 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:01,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:01,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:01,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:01,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:01,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:01,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:01,924 DEBUG [Thread-1723] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:01,924 DEBUG [Thread-1723] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:01,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:01,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:01,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:01,970 DEBUG [Thread-1724] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:56827 2023-03-28 20:55:01,970 DEBUG [Thread-1724] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:01,971 DEBUG [Thread-1732] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45de5ad8 to 127.0.0.1:56827 2023-03-28 20:55:01,971 DEBUG [Thread-1732] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:01,971 DEBUG [Thread-1730] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b4fa5d3 to 127.0.0.1:56827 2023-03-28 20:55:01,971 DEBUG [Thread-1730] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:01,973 DEBUG [Thread-1728] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:56827 2023-03-28 20:55:01,973 DEBUG [Thread-1728] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:01,973 DEBUG [Thread-1726] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:56827 2023-03-28 20:55:01,973 DEBUG [Thread-1726] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:02,024 DEBUG [Thread-1723] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:02,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036962098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036962098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c2351e8d444e448e8cff485089ff97d2 2023-03-28 20:55:02,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46520 deadline: 1680036962401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680036962402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46598 deadline: 1680036962488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46560 deadline: 1680036962489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:02,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/66621faacfa146fc937785f4c99e2f2d 2023-03-28 20:55:02,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/2332628e33f941f6a1f81828c68fbd74 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74 2023-03-28 20:55:02,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74, entries=250, sequenceid=680, filesize=16.7 K 2023-03-28 20:55:02,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/c2351e8d444e448e8cff485089ff97d2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2 2023-03-28 20:55:02,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2, entries=150, sequenceid=680, filesize=11.9 K 2023-03-28 20:55:02,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/66621faacfa146fc937785f4c99e2f2d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d 2023-03-28 20:55:02,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d, entries=150, sequenceid=680, filesize=11.9 K 2023-03-28 20:55:02,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16a0b6726aa26da22f14d3e4178de945 in 845ms, sequenceid=680, compaction requested=true 2023-03-28 20:55:02,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:02,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:02,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:02,595 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:02,595 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:02,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:02,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:02,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:02,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/B is initiating minor compaction (all files) 2023-03-28 20:55:02,596 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/A is initiating minor compaction (all files) 2023-03-28 20:55:02,597 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/B in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:02,597 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/A in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:02,597 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/19d2949fea734918971f120d0efbb5c6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=37.2 K 2023-03-28 20:55:02,597 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/46ea418b76c44ca48ab7a6dd7c0af736, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=42.0 K 2023-03-28 20:55:02,597 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 19d2949fea734918971f120d0efbb5c6, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:02,597 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 46ea418b76c44ca48ab7a6dd7c0af736, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:02,597 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b337b52c1cba46fab288dddc4561c4f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680036900343 2023-03-28 20:55:02,598 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ded0b39fc19d48b981fb66207ce6be2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680036900343 2023-03-28 20:55:02,598 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c2351e8d444e448e8cff485089ff97d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680036901436 2023-03-28 20:55:02,598 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 2332628e33f941f6a1f81828c68fbd74, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680036901430 2023-03-28 20:55:02,604 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#B#compaction#358 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:02,605 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#A#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:02,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=114.05 KB heapSize=339.89 KB 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:02,905 DEBUG [Thread-1715] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:56827 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:02,905 DEBUG [Thread-1715] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:02,905 DEBUG [Thread-1719] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3562c7bf to 127.0.0.1:56827 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:02,905 DEBUG [Thread-1719] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:02,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:03,016 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/c3aa7c61a7f447b9869f28d13721367a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c3aa7c61a7f447b9869f28d13721367a 2023-03-28 20:55:03,016 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/e4e616eee351480e84b8515eba362695 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e4e616eee351480e84b8515eba362695 2023-03-28 20:55:03,020 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/B of 16a0b6726aa26da22f14d3e4178de945 into e4e616eee351480e84b8515eba362695(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:03,020 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/A of 16a0b6726aa26da22f14d3e4178de945 into c3aa7c61a7f447b9869f28d13721367a(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:03,020 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:03,020 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:03,020 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/B, priority=13, startTime=1680036902595; duration=0sec 2023-03-28 20:55:03,020 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/A, priority=13, startTime=1680036902595; duration=0sec 2023-03-28 20:55:03,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:03,021 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:03,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:03,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:03,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 16a0b6726aa26da22f14d3e4178de945/C is initiating minor compaction (all files) 2023-03-28 20:55:03,022 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16a0b6726aa26da22f14d3e4178de945/C in TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:03,022 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eec861018819451c89f5e8ec485bc6e8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp, totalSize=37.2 K 2023-03-28 20:55:03,022 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting eec861018819451c89f5e8ec485bc6e8, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680036900286 2023-03-28 20:55:03,022 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3a3e3df84f434004ab6506470fc440a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680036900343 2023-03-28 20:55:03,022 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 66621faacfa146fc937785f4c99e2f2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680036901436 2023-03-28 20:55:03,027 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 16a0b6726aa26da22f14d3e4178de945#C#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:03,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ae032157aa244fad8b5aa0de6a6d9fe8 2023-03-28 20:55:03,434 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/cf27c8a1c1be4cb2b15bb7934b8e43ed as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cf27c8a1c1be4cb2b15bb7934b8e43ed 2023-03-28 20:55:03,437 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16a0b6726aa26da22f14d3e4178de945/C of 16a0b6726aa26da22f14d3e4178de945 into cf27c8a1c1be4cb2b15bb7934b8e43ed(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:03,437 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:03,437 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945., storeName=16a0b6726aa26da22f14d3e4178de945/C, priority=13, startTime=1680036902595; duration=0sec 2023-03-28 20:55:03,437 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:03,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f3adda6c94d64ae882d5bc3f8c13258e 2023-03-28 20:55:04,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/d8438d4f57d34b51b7ec4f67bc619a01 2023-03-28 20:55:04,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/ae032157aa244fad8b5aa0de6a6d9fe8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ae032157aa244fad8b5aa0de6a6d9fe8 2023-03-28 20:55:04,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ae032157aa244fad8b5aa0de6a6d9fe8, entries=150, sequenceid=701, filesize=11.9 K 2023-03-28 20:55:04,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/f3adda6c94d64ae882d5bc3f8c13258e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f3adda6c94d64ae882d5bc3f8c13258e 2023-03-28 20:55:04,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f3adda6c94d64ae882d5bc3f8c13258e, entries=150, sequenceid=701, filesize=11.9 K 2023-03-28 20:55:04,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/d8438d4f57d34b51b7ec4f67bc619a01 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/d8438d4f57d34b51b7ec4f67bc619a01 2023-03-28 20:55:04,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/d8438d4f57d34b51b7ec4f67bc619a01, entries=150, sequenceid=701, filesize=11.9 K 2023-03-28 20:55:04,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=0 B/0 for 16a0b6726aa26da22f14d3e4178de945 in 1236ms, sequenceid=701, compaction requested=false 2023-03-28 20:55:04,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:04,491 DEBUG [Thread-1713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:56827 2023-03-28 20:55:04,491 DEBUG [Thread-1713] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:04,500 DEBUG [Thread-1717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:56827 2023-03-28 20:55:04,500 DEBUG [Thread-1717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:07,521 DEBUG [Thread-1721] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6705d115 to 127.0.0.1:56827 2023-03-28 20:55:07,521 DEBUG [Thread-1721] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 218 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:55:07,521 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5270 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 4999 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 4966 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5248 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5020 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:55:07,522 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:55:07,522 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:56827 2023-03-28 20:55:07,522 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:07,523 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:55:07,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:55:07,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:07,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:07,526 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036907526"}]},"ts":"1680036907526"} 2023-03-28 20:55:07,527 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:55:07,529 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:55:07,530 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, UNASSIGN}] 2023-03-28 20:55:07,531 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, UNASSIGN 2023-03-28 20:55:07,531 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=16a0b6726aa26da22f14d3e4178de945, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:07,531 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036907531"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036907531"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036907531"}]},"ts":"1680036907531"} 2023-03-28 20:55:07,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:07,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:07,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 16a0b6726aa26da22f14d3e4178de945, disabling compactions & flushes 2023-03-28 20:55:07,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. after waiting 0 ms 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:07,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 16a0b6726aa26da22f14d3e4178de945 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=A 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=B 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16a0b6726aa26da22f14d3e4178de945, store=C 2023-03-28 20:55:07,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:07,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:08,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1bf3e947b4ae49bca92ed2be9ae9f283 2023-03-28 20:55:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:08,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/616ce4f544e24d25a4936a28f88cc908 2023-03-28 20:55:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:08,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/c146c9d48b4640408998892055e27db2 2023-03-28 20:55:08,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/A/1bf3e947b4ae49bca92ed2be9ae9f283 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bf3e947b4ae49bca92ed2be9ae9f283 2023-03-28 20:55:08,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bf3e947b4ae49bca92ed2be9ae9f283, entries=150, sequenceid=710, filesize=11.9 K 2023-03-28 20:55:08,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/B/616ce4f544e24d25a4936a28f88cc908 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/616ce4f544e24d25a4936a28f88cc908 2023-03-28 20:55:08,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/616ce4f544e24d25a4936a28f88cc908, entries=150, sequenceid=710, filesize=11.9 K 2023-03-28 20:55:08,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/.tmp/C/c146c9d48b4640408998892055e27db2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/c146c9d48b4640408998892055e27db2 2023-03-28 20:55:08,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/c146c9d48b4640408998892055e27db2, entries=150, sequenceid=710, filesize=11.9 K 2023-03-28 20:55:08,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 16a0b6726aa26da22f14d3e4178de945 in 1236ms, sequenceid=710, compaction requested=true 2023-03-28 20:55:08,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/70e7f91e669a498b99ccaf64617225fc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/cde54d8920dd4620add4e285209fce34, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/45d4db048e6a4c49ab3ae438a122b489, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33e397ba6b9f42dc8a87d3542af71334, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a94a28b4d0ee415eaf346b0559cd0315, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f8cea17974a0478bb3ac75304b88e33c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/4c9f9aea92434599b884f73a413df07c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c22538478c8a431596887946bacdf93f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bb1a7b30e28442b9e045e4ef3594a5e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0a7e2e7595af42eb81cdbc3b1353655c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/46ea418b76c44ca48ab7a6dd7c0af736, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74] to archive 2023-03-28 20:55:08,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:08,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7fb9089dd4e64e1f8cea629027635f1a 2023-03-28 20:55:08,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/074454af429945719ae9cc1c7f8996e6 2023-03-28 20:55:08,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0de23fb71ece4734855b9481f78abb50 2023-03-28 20:55:08,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/96a7423b2b3a43d790a154f097d1637c 2023-03-28 20:55:08,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e4865dba7832479e9639a09bddc4a6a8 2023-03-28 20:55:08,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/7a4a05e70ded4b5ba771e4c8f2e12b2c 2023-03-28 20:55:08,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a6b8fee1168f4b20bb7df180f72a910a 2023-03-28 20:55:08,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/85cd5a1b6bc44c72b8feb0ae85da1a06 2023-03-28 20:55:08,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/39b46f42b1d041a89cb0c1f4efcb0b9d 2023-03-28 20:55:08,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/70e7f91e669a498b99ccaf64617225fc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/70e7f91e669a498b99ccaf64617225fc 2023-03-28 20:55:08,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33c313adaf4d4a2cbb8cacd854168115 2023-03-28 20:55:08,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a05c2cb16a284420a2f44b5fe96baf57 2023-03-28 20:55:08,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1d1587c7cad944e9b8f8a6b90ab02288 2023-03-28 20:55:08,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/cde54d8920dd4620add4e285209fce34 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/cde54d8920dd4620add4e285209fce34 2023-03-28 20:55:08,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/b32a4cef629a40578ab74a3aae30fb87 2023-03-28 20:55:08,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/5f5e040d06904a39920ca6f04ab6c8ad 2023-03-28 20:55:08,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/45d4db048e6a4c49ab3ae438a122b489 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/45d4db048e6a4c49ab3ae438a122b489 2023-03-28 20:55:08,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8de3c65e6cc447109ff47375dcb04e82 2023-03-28 20:55:08,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/8fae27fe06104cf78c1e2203d2a90754 2023-03-28 20:55:08,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ca2d5ca30dc44af4b89c9979bf8be287 2023-03-28 20:55:08,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33e397ba6b9f42dc8a87d3542af71334 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/33e397ba6b9f42dc8a87d3542af71334 2023-03-28 20:55:08,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/aa5b82e5d19441c1b1af461909c40587 2023-03-28 20:55:08,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ac1f8bc34b09416cab71a894422f67d5 2023-03-28 20:55:08,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c988989f510f4f78950bbd54fa37cb12 2023-03-28 20:55:08,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/56d8417312e9491a9534c4f8cbd41640 2023-03-28 20:55:08,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a94a28b4d0ee415eaf346b0559cd0315 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a94a28b4d0ee415eaf346b0559cd0315 2023-03-28 20:55:08,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0fcd79eb0e51451891f56d63151397d5 2023-03-28 20:55:08,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/53acc507110442749c0e67b1a6c22c6c 2023-03-28 20:55:08,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/e250ab4c406340c5bdcda338cea9ca0d 2023-03-28 20:55:08,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f8cea17974a0478bb3ac75304b88e33c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f8cea17974a0478bb3ac75304b88e33c 2023-03-28 20:55:08,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1cae8b92413d4de79df470cd847af57b 2023-03-28 20:55:08,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/262ef59ea6bb4851af8f4f7f3ceb24da 2023-03-28 20:55:08,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/4c9f9aea92434599b884f73a413df07c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/4c9f9aea92434599b884f73a413df07c 2023-03-28 20:55:08,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/a3fd3c456571467189226c886eaf8686 2023-03-28 20:55:08,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/af7153d50fe34808845c958db3f19b03 2023-03-28 20:55:08,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c22538478c8a431596887946bacdf93f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c22538478c8a431596887946bacdf93f 2023-03-28 20:55:08,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/54cc3e141d614b72bc69297a5f5879cc 2023-03-28 20:55:08,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bb1a7b30e28442b9e045e4ef3594a5e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bb1a7b30e28442b9e045e4ef3594a5e 2023-03-28 20:55:08,976 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/66dc1774d57a40e4b3f132baa2298ec3 2023-03-28 20:55:08,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/f903d8e7bd774600a62066d791fc4d0a 2023-03-28 20:55:08,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/df23346cc70841dfa9ffb0f411aef56e 2023-03-28 20:55:08,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0a7e2e7595af42eb81cdbc3b1353655c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/0a7e2e7595af42eb81cdbc3b1353655c 2023-03-28 20:55:08,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/fc938e629c6c47488f8d1be1d9a7f5e5 2023-03-28 20:55:08,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/eebd5411982749d1ab55f783ff76f207 2023-03-28 20:55:08,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/46ea418b76c44ca48ab7a6dd7c0af736 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/46ea418b76c44ca48ab7a6dd7c0af736 2023-03-28 20:55:08,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ec26fe0428cf4d68bcea5f46bacea8a0 2023-03-28 20:55:08,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ded0b39fc19d48b981fb66207ce6be2c 2023-03-28 20:55:08,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/2332628e33f941f6a1f81828c68fbd74 2023-03-28 20:55:08,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b22f2d3fba044c85821f940f6cb6cc4c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/26a47550a72c4e5c9fba17d36a24bf6b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/71dad285676844ebb7c2dfd91be1599d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/9f3552a5b46140b583a5b0ce8c28b3bb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/5b2d57ef3d31467f8f55204e1ea332e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/83c09c9032054408b3750c61f0dba7f7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/6e9385b71d0a452b892cde4bf42d52f9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7855c0e288d44c5786784667027f1f4c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a3f8c253ddb34290aae015e49e534b2c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/38dcb4f2aabb452c804b8a759890380d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/19d2949fea734918971f120d0efbb5c6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2] to archive 2023-03-28 20:55:08,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:08,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3d4439884d54459934a6c8ba78c6707 2023-03-28 20:55:08,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2f800e74bdda44c0923fe7b8f6848c5d 2023-03-28 20:55:08,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/2d6a7e534bdf4547834726088499f29c 2023-03-28 20:55:08,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/18ed36948e4e4f12aaf14c612cb09310 2023-03-28 20:55:08,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a33b5849eec84895bbcdd5db6802455e 2023-03-28 20:55:08,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e8aad2e087844eaab1eb1a9301b94f02 2023-03-28 20:55:08,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ac457b1970ca4a94aa650980c797985b 2023-03-28 20:55:08,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c8edf60502374289a61428c2a1e2790d 2023-03-28 20:55:08,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b22f2d3fba044c85821f940f6cb6cc4c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b22f2d3fba044c85821f940f6cb6cc4c 2023-03-28 20:55:08,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a03a41b2d1c14d86923dfe714ec3ab05 2023-03-28 20:55:08,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/3b9cb6618e024b82ad273925d7472b46 2023-03-28 20:55:08,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/8c856d391c20470195cd0dec2679c48f 2023-03-28 20:55:09,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1d7bbcf72cea43b9aa625de521137e35 2023-03-28 20:55:09,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/26a47550a72c4e5c9fba17d36a24bf6b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/26a47550a72c4e5c9fba17d36a24bf6b 2023-03-28 20:55:09,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f26ebc9c43094b048353941a7c0642b3 2023-03-28 20:55:09,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7dd4099b616b4ea9b92295a94c6d87de 2023-03-28 20:55:09,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/71dad285676844ebb7c2dfd91be1599d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/71dad285676844ebb7c2dfd91be1599d 2023-03-28 20:55:09,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fade0e9c451b41829efa13d749b4d88f 2023-03-28 20:55:09,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/09c5dfbe39bf4bc495344ab4182e09b3 2023-03-28 20:55:09,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/9f3552a5b46140b583a5b0ce8c28b3bb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/9f3552a5b46140b583a5b0ce8c28b3bb 2023-03-28 20:55:09,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c51aef9d703e46aeb3dd7ba653c81926 2023-03-28 20:55:09,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c3672cd8b8a5425c9e87a6cad62fec60 2023-03-28 20:55:09,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/07e95d0576ab4fa19b084e14c154b89b 2023-03-28 20:55:09,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/ab2bc385243744eabc1f3e1e85b126b9 2023-03-28 20:55:09,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/5b2d57ef3d31467f8f55204e1ea332e9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/5b2d57ef3d31467f8f55204e1ea332e9 2023-03-28 20:55:09,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/cb123333a6414403b924fff1caa1de2a 2023-03-28 20:55:09,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e601a3da7ebb400f92ff0cefa112f98a 2023-03-28 20:55:09,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c075f91e959d4b09bf4bc4811d1aae14 2023-03-28 20:55:09,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/83c09c9032054408b3750c61f0dba7f7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/83c09c9032054408b3750c61f0dba7f7 2023-03-28 20:55:09,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f65d683affc140a8867f0a91e2321a3f 2023-03-28 20:55:09,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/59b25545e1794a11961287b7331084d1 2023-03-28 20:55:09,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/6e9385b71d0a452b892cde4bf42d52f9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/6e9385b71d0a452b892cde4bf42d52f9 2023-03-28 20:55:09,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a77b7d948a7b40b7a22862eea951623b 2023-03-28 20:55:09,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/fb8096f9b47e4f449881e3faa298a336 2023-03-28 20:55:09,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7855c0e288d44c5786784667027f1f4c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/7855c0e288d44c5786784667027f1f4c 2023-03-28 20:55:09,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/feeabe1e2d2e458cb44e5bdd27bce213 2023-03-28 20:55:09,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c9702f027fcd4ce79bd306636029776e 2023-03-28 20:55:09,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a3f8c253ddb34290aae015e49e534b2c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/a3f8c253ddb34290aae015e49e534b2c 2023-03-28 20:55:09,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/1c1f552d80ec4878a6b2d6c75d03aeef 2023-03-28 20:55:09,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/75e7d84cafab4435baa5ca047d54d791 2023-03-28 20:55:09,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/38dcb4f2aabb452c804b8a759890380d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/38dcb4f2aabb452c804b8a759890380d 2023-03-28 20:55:09,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/0acf20e55ebd47e59e531a33fbd75b44 2023-03-28 20:55:09,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e0bd61f144f848cbbe188d8e793666eb 2023-03-28 20:55:09,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c1fce4f8b2744fcd853313f7316ae4c6 2023-03-28 20:55:09,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/19d2949fea734918971f120d0efbb5c6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/19d2949fea734918971f120d0efbb5c6 2023-03-28 20:55:09,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/03548d4d7d234358b3e67579deeab118 2023-03-28 20:55:09,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/b337b52c1cba46fab288dddc4561c4f8 2023-03-28 20:55:09,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/c2351e8d444e448e8cff485089ff97d2 2023-03-28 20:55:09,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/321b3fb532b142ebb5e7ad7309e80bd8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/277dd6fec2c74697a29e18726998b304, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/25fd3c979fa249619ff9681f1691d8b7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1ca9977218bb4168abd8fd4a65ce0c4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/6aab56a649ad435cb6e9e8e190590273, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/63a68a697e8e40febfc34d6c9e2016fb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3fff8b138dd4cdc90e719dac4509a66, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ff2d29d8bbd34308a74da478e5fa969f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/9b8c2de408b84094830986dfa1f094f2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eec861018819451c89f5e8ec485bc6e8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d] to archive 2023-03-28 20:55:09,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:09,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ecd433ba2beb4cafb151cbc42774a60b 2023-03-28 20:55:09,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a33258c3bc9d4af69eaaf9c8cb0e5100 2023-03-28 20:55:09,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a808d7c8942b4f17a1c57ae05daeb358 2023-03-28 20:55:09,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8cd91d32d3d24944955b233fb5e01eff 2023-03-28 20:55:09,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3e97218879244cdb6d3cb3a1e60fdf0 2023-03-28 20:55:09,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4add4fa809964ce3abc3b30f9d1ee4d6 2023-03-28 20:55:09,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4a1494b2a5c84c01a84165b85f3c43d1 2023-03-28 20:55:09,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cb1a1b9da42b471d9731a3d53d2dc1af 2023-03-28 20:55:09,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/321b3fb532b142ebb5e7ad7309e80bd8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/321b3fb532b142ebb5e7ad7309e80bd8 2023-03-28 20:55:09,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/dc432f52da5743d0a1926f5c0afc9cc3 2023-03-28 20:55:09,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eb4d75bad6ae45748d3de00132d1198e 2023-03-28 20:55:09,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/77c240d652ca49bf978950365b250564 2023-03-28 20:55:09,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/f03dc6ae09074ebcac498b265c22fcdb 2023-03-28 20:55:09,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/277dd6fec2c74697a29e18726998b304 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/277dd6fec2c74697a29e18726998b304 2023-03-28 20:55:09,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/202ae780ee284f9ca13cf8111a6ee31b 2023-03-28 20:55:09,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ae44b79039d24dc5bcf2305da4c23f1d 2023-03-28 20:55:09,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/25fd3c979fa249619ff9681f1691d8b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/25fd3c979fa249619ff9681f1691d8b7 2023-03-28 20:55:09,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/638c3396c5a74c17a7e7d44c174ab9b5 2023-03-28 20:55:09,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/50eb1a3e5fa04e809ab4ab481a85547f 2023-03-28 20:55:09,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1ca9977218bb4168abd8fd4a65ce0c4a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1ca9977218bb4168abd8fd4a65ce0c4a 2023-03-28 20:55:09,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ea52078f81964068a8fc28be676cf930 2023-03-28 20:55:09,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/22d7dab36c6d497f9645183397d4882d 2023-03-28 20:55:09,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4f6a31938780415d9a61a5d962855a12 2023-03-28 20:55:09,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3e6909ebc3b0460e87f8003fa9cf0276 2023-03-28 20:55:09,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/6aab56a649ad435cb6e9e8e190590273 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/6aab56a649ad435cb6e9e8e190590273 2023-03-28 20:55:09,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a926e23988144743b476aa2d72e13dbc 2023-03-28 20:55:09,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/8a5d12bc508b40dda820b2b6a86a2fab 2023-03-28 20:55:09,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/0c1fe4b8fe954011b6aa8bd6d01b315a 2023-03-28 20:55:09,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/4c8abf8e9173494daa57ce5841bc7215 2023-03-28 20:55:09,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/63a68a697e8e40febfc34d6c9e2016fb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/63a68a697e8e40febfc34d6c9e2016fb 2023-03-28 20:55:09,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e7289172e16047ba90995f6ac77257d3 2023-03-28 20:55:09,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/e1d47af6cd45428f9b47d9a1b48958fc 2023-03-28 20:55:09,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/efbb916d71604942a2191592b65af107 2023-03-28 20:55:09,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3fff8b138dd4cdc90e719dac4509a66 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a3fff8b138dd4cdc90e719dac4509a66 2023-03-28 20:55:09,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ccde2cdfa2d64591bd9333e5072fddea 2023-03-28 20:55:09,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/de2925cb3fe84b55972c1c761000fad4 2023-03-28 20:55:09,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ff2d29d8bbd34308a74da478e5fa969f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/ff2d29d8bbd34308a74da478e5fa969f 2023-03-28 20:55:09,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/1451b1a835414ec0addff636222b18b6 2023-03-28 20:55:09,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3935402521604a249c3fccc50230da2e 2023-03-28 20:55:09,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/9b8c2de408b84094830986dfa1f094f2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/9b8c2de408b84094830986dfa1f094f2 2023-03-28 20:55:09,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/a0bae98a4cbc4ce9a1cf18ffb6a04c70 2023-03-28 20:55:09,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/614a5a89275c400e83555e48f7841c38 2023-03-28 20:55:09,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3ffb609a8cd64049ba51a587e7a2a267 2023-03-28 20:55:09,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eec861018819451c89f5e8ec485bc6e8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/eec861018819451c89f5e8ec485bc6e8 2023-03-28 20:55:09,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/80d968de18304b918687baa45fc1d8c1 2023-03-28 20:55:09,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/3a3e3df84f434004ab6506470fc440a5 2023-03-28 20:55:09,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/66621faacfa146fc937785f4c99e2f2d 2023-03-28 20:55:09,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/recovered.edits/713.seqid, newMaxSeqId=713, maxSeqId=1 2023-03-28 20:55:09,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945. 2023-03-28 20:55:09,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 16a0b6726aa26da22f14d3e4178de945: 2023-03-28 20:55:09,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:09,090 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=16a0b6726aa26da22f14d3e4178de945, regionState=CLOSED 2023-03-28 20:55:09,090 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036909090"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036909090"}]},"ts":"1680036909090"} 2023-03-28 20:55:09,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-28 20:55:09,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 16a0b6726aa26da22f14d3e4178de945, server=jenkins-hbase4.apache.org,42693,1680036815956 in 1.5590 sec 2023-03-28 20:55:09,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-28 20:55:09,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16a0b6726aa26da22f14d3e4178de945, UNASSIGN in 1.5640 sec 2023-03-28 20:55:09,095 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036909095"}]},"ts":"1680036909095"} 2023-03-28 20:55:09,096 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:55:09,097 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:55:09,099 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5750 sec 2023-03-28 20:55:09,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-28 20:55:09,630 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-28 20:55:09,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:55:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,633 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-28 20:55:09,634 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,638 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:09,641 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/recovered.edits] 2023-03-28 20:55:09,644 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bf3e947b4ae49bca92ed2be9ae9f283 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/1bf3e947b4ae49bca92ed2be9ae9f283 2023-03-28 20:55:09,645 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ae032157aa244fad8b5aa0de6a6d9fe8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/ae032157aa244fad8b5aa0de6a6d9fe8 2023-03-28 20:55:09,646 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c3aa7c61a7f447b9869f28d13721367a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/A/c3aa7c61a7f447b9869f28d13721367a 2023-03-28 20:55:09,648 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/616ce4f544e24d25a4936a28f88cc908 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/616ce4f544e24d25a4936a28f88cc908 2023-03-28 20:55:09,649 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e4e616eee351480e84b8515eba362695 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/e4e616eee351480e84b8515eba362695 2023-03-28 20:55:09,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f3adda6c94d64ae882d5bc3f8c13258e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/B/f3adda6c94d64ae882d5bc3f8c13258e 2023-03-28 20:55:09,654 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/c146c9d48b4640408998892055e27db2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/c146c9d48b4640408998892055e27db2 2023-03-28 20:55:09,656 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cf27c8a1c1be4cb2b15bb7934b8e43ed to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/cf27c8a1c1be4cb2b15bb7934b8e43ed 2023-03-28 20:55:09,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/d8438d4f57d34b51b7ec4f67bc619a01 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/C/d8438d4f57d34b51b7ec4f67bc619a01 2023-03-28 20:55:09,660 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/recovered.edits/713.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945/recovered.edits/713.seqid 2023-03-28 20:55:09,661 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/16a0b6726aa26da22f14d3e4178de945 2023-03-28 20:55:09,661 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:55:09,663 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,664 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:55:09,666 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:55:09,667 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,667 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:55:09,668 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036909667"}]},"ts":"9223372036854775807"} 2023-03-28 20:55:09,676 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:55:09,676 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 16a0b6726aa26da22f14d3e4178de945, NAME => 'TestAcidGuarantees,,1680036879793.16a0b6726aa26da22f14d3e4178de945.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:55:09,676 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:55:09,676 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036909676"}]},"ts":"9223372036854775807"} 2023-03-28 20:55:09,679 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:55:09,681 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 50 msec 2023-03-28 20:55:09,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-28 20:55:09,735 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-28 20:55:09,749 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225 (was 231), OpenFileDescriptor=589 (was 576) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 360), ProcessCount=171 (was 171), AvailableMemoryMB=6609 (was 6787) 2023-03-28 20:55:09,763 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=225, OpenFileDescriptor=589, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=171, AvailableMemoryMB=6608 2023-03-28 20:55:09,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:55:09,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:55:09,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:09,772 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:55:09,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-28 20:55:09,773 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:55:09,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-28 20:55:09,774 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:09,775 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 empty. 2023-03-28 20:55:09,775 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:09,775 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:55:09,784 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:55:09,787 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1d7220861206a2835f715e6061645bc2, NAME => 'TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:55:09,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-28 20:55:10,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-28 20:55:10,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:55:10,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1d7220861206a2835f715e6061645bc2, disabling compactions & flushes 2023-03-28 20:55:10,196 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. after waiting 0 ms 2023-03-28 20:55:10,196 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,196 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,197 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:10,199 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:55:10,199 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036910199"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036910199"}]},"ts":"1680036910199"} 2023-03-28 20:55:10,200 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:55:10,201 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:55:10,201 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036910201"}]},"ts":"1680036910201"} 2023-03-28 20:55:10,202 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:55:10,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, ASSIGN}] 2023-03-28 20:55:10,207 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, ASSIGN 2023-03-28 20:55:10,208 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:55:10,359 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:10,359 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036910359"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036910359"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036910359"}]},"ts":"1680036910359"} 2023-03-28 20:55:10,360 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:10,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-28 20:55:10,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1d7220861206a2835f715e6061645bc2, NAME => 'TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:55:10,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:55:10,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,516 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,517 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A 2023-03-28 20:55:10,517 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A 2023-03-28 20:55:10,517 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:10,518 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName A 2023-03-28 20:55:10,518 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:10,518 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,519 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B 2023-03-28 20:55:10,519 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B 2023-03-28 20:55:10,519 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:10,519 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName B 2023-03-28 20:55:10,520 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:10,520 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,521 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C 2023-03-28 20:55:10,521 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C 2023-03-28 20:55:10,521 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:10,521 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName C 2023-03-28 20:55:10,521 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:10,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:55:10,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:10,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:55:10,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1d7220861206a2835f715e6061645bc2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71753418, jitterRate=0.06920924782752991}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:55:10,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:10,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., pid=37, masterSystemTime=1680036910512 2023-03-28 20:55:10,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:10,528 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:10,528 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036910528"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036910528"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036910528"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036910528"}]},"ts":"1680036910528"} 2023-03-28 20:55:10,530 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-28 20:55:10,530 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 in 169 msec 2023-03-28 20:55:10,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-28 20:55:10,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, ASSIGN in 324 msec 2023-03-28 20:55:10,531 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:55:10,532 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036910532"}]},"ts":"1680036910532"} 2023-03-28 20:55:10,532 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:55:10,534 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:55:10,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 768 msec 2023-03-28 20:55:10,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-28 20:55:10,876 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-28 20:55:10,877 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x67b6d133 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:10,882 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@494529f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:10,883 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:10,884 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:10,885 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:55:10,886 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58238, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:55:10,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:55:10,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:55:10,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:11,297 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:55:11,298 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:55:11,298 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:55:11,300 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-28 20:55:11,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, REOPEN/MOVE}] 2023-03-28 20:55:11,303 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, REOPEN/MOVE 2023-03-28 20:55:11,303 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,304 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036911303"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036911303"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036911303"}]},"ts":"1680036911303"} 2023-03-28 20:55:11,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:11,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1d7220861206a2835f715e6061645bc2, disabling compactions & flushes 2023-03-28 20:55:11,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. after waiting 0 ms 2023-03-28 20:55:11,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-28 20:55:11,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:11,460 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1d7220861206a2835f715e6061645bc2 to self. 2023-03-28 20:55:11,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,462 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=CLOSED 2023-03-28 20:55:11,462 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036911462"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036911462"}]},"ts":"1680036911462"} 2023-03-28 20:55:11,464 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-28 20:55:11,464 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 in 159 msec 2023-03-28 20:55:11,464 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=true 2023-03-28 20:55:11,615 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,615 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036911615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036911615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036911615"}]},"ts":"1680036911615"} 2023-03-28 20:55:11,616 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:11,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1d7220861206a2835f715e6061645bc2, NAME => 'TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:55:11,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:55:11,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,773 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,774 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A 2023-03-28 20:55:11,774 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A 2023-03-28 20:55:11,774 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:11,775 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName A 2023-03-28 20:55:11,776 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:11,776 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,777 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B 2023-03-28 20:55:11,777 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B 2023-03-28 20:55:11,777 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:11,777 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName B 2023-03-28 20:55:11,778 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:11,778 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,779 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C 2023-03-28 20:55:11,779 DEBUG [StoreOpener-1d7220861206a2835f715e6061645bc2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C 2023-03-28 20:55:11,779 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:11,779 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1d7220861206a2835f715e6061645bc2 columnFamilyName C 2023-03-28 20:55:11,780 INFO [StoreOpener-1d7220861206a2835f715e6061645bc2-1] regionserver.HStore(310): Store=1d7220861206a2835f715e6061645bc2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:11,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:55:11,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1d7220861206a2835f715e6061645bc2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75101645, jitterRate=0.11910171806812286}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:55:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:11,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., pid=42, masterSystemTime=1680036911767 2023-03-28 20:55:11,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,787 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,787 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036911787"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036911787"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036911787"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036911787"}]},"ts":"1680036911787"} 2023-03-28 20:55:11,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-28 20:55:11,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 in 171 msec 2023-03-28 20:55:11,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-28 20:55:11,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, REOPEN/MOVE in 486 msec 2023-03-28 20:55:11,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-28 20:55:11,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-03-28 20:55:11,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-03-28 20:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-28 20:55:11,794 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1cee66b5 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,799 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d800352, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,799 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x4c1d19b3 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,811 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2329054b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,811 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x14e151fb to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,817 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30951255, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,817 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x455a88b1 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,820 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5baa6768, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,821 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x722b2407 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,824 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63a535f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,824 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x2d4bfeca to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,829 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@678b4c3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,829 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x313c7ad6 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,832 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43736f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,833 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x20d28b68 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,838 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72eaaf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,838 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x750d2a7d to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,841 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4130667f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,842 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x27313b38 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:11,846 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b73a361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:11,855 DEBUG [hconnection-0x59a7a632-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,855 DEBUG [hconnection-0x178c504-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,857 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,857 DEBUG [hconnection-0x741e00d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:11,858 DEBUG [hconnection-0x29a45885-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,858 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48240, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,860 DEBUG [hconnection-0x292a2e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,860 DEBUG [hconnection-0x71dc531b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,865 DEBUG [hconnection-0x36399ef7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,865 DEBUG [hconnection-0x6b57a4e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,865 DEBUG [hconnection-0x3fe1478a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,866 DEBUG [hconnection-0x13d765b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:11,866 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48224, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,866 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,866 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48264, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,866 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48252, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,868 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,868 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48286, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,868 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48296, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,871 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48278, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:11,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:11,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:11,875 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:11,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:11,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:11,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:11,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:11,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:11,879 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:11,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:11,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:11,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:11,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:11,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:11,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:11,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:11,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:11,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:11,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:11,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:11,894 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3e4f7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:11,896 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:11,896 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,897 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:11,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:11,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:11,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036971895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:11,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036971896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:11,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036971898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:11,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:11,902 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:11,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:11,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:11,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:11,902 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:11,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:11,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036971899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:11,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036971900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:11,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:11,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:11,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,908 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,910 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:11,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:11,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:11,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:11,912 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:11,912 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:11,911 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:11,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:11,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036971999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036971999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036972002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036972004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036972005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,012 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,016 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,029 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,030 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650db17c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,032 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,032 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,033 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,035 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,035 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,037 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,038 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,038 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,039 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,040 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,042 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,042 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,042 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,143 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,148 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,150 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,150 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9b0b89[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,161 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,161 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,167 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,171 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,172 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,173 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,173 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,173 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,181 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,181 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036972202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036972204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036972206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036972208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036972209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,281 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,284 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,285 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@de71e37[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,294 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,297 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,297 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,299 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,314 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,314 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,314 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303280c673c2794fc4404960dbccc7be93b52 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280c673c2794fc4404960dbccc7be93b52 2023-03-28 20:55:12,414 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,417 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,430 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6789c720[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,436 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,438 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,438 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,440 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,440 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,440 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,444 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,445 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,445 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,445 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,445 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036972508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036972510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036972514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036972517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:12,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036972517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,545 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,549 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,556 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,556 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,556 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,556 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f070cfc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,560 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,560 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,563 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,565 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,565 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,573 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,573 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,573 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,673 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,677 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,678 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,679 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,680 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e795d6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,684 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,684 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,690 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,690 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,694 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,698 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,698 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,698 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,719 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/ff500f6f854040c4b6159f15656ec5fd 2023-03-28 20:55:12,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/497afdf67a8e47eb8a8325fc6b2749b8 2023-03-28 20:55:12,798 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,801 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,804 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc43c5b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,810 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,810 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,810 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,815 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,817 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,817 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,818 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,819 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,819 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,819 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,819 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,823 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:12,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,823 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,824 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,824 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:12,924 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:12,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:12,927 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:12,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:12,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:12,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,929 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,930 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:12,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:12,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:12,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:12,931 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,931 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:12,931 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:12,931 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:12,934 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54dd5404[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,937 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:12,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:12,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:12,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:12,938 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,938 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:12,938 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,938 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:12,941 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:12,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:12,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:12,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:12,941 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:12,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:12,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,943 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:12,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,944 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:12,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:12,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:12,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:12,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:12,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:12,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:12,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:12,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:12,949 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:12,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:12,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:12,949 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:13,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:13,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036973013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:13,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036973015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036973020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036973020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036973025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,049 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:13,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:13,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:13,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:13,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:13,052 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:13,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:13,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:13,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,054 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:13,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:13,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:13,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:13,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:13,056 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:13,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad28f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:13,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:13,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:13,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:13,066 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:13,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:13,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:13,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:13,066 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:13,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:13,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:13,074 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,074 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,074 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/bda98b77a3e64ea19ad21bd7ae9dab96 2023-03-28 20:55:13,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/ff500f6f854040c4b6159f15656ec5fd as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd 2023-03-28 20:55:13,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd, entries=200, sequenceid=17, filesize=31.9 K 2023-03-28 20:55:13,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/497afdf67a8e47eb8a8325fc6b2749b8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8 2023-03-28 20:55:13,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:55:13,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/bda98b77a3e64ea19ad21bd7ae9dab96 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96 2023-03-28 20:55:13,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:55:13,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 1d7220861206a2835f715e6061645bc2 in 1280ms, sequenceid=17, compaction requested=false 2023-03-28 20:55:13,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:13,174 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:13,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:13,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:13,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:13,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:13,177 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:13,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:13,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:13,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:13,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:13,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:13,181 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:13,182 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:13,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:13,594 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328d3b16dbc9d384a0c98ffbf14d2cf1e32 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d3b16dbc9d384a0c98ffbf14d2cf1e32 2023-03-28 20:55:13,605 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/ea98ca487a4a48f1b2fa40c45747deb8 2023-03-28 20:55:13,632 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/4ed6912c8bef429ebdf1126f82115553 2023-03-28 20:55:13,662 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c39ece258f6d4e2ba19093b8d8052b5f 2023-03-28 20:55:13,667 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/ea98ca487a4a48f1b2fa40c45747deb8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8 2023-03-28 20:55:13,670 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8, entries=150, sequenceid=37, filesize=25.1 K 2023-03-28 20:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/4ed6912c8bef429ebdf1126f82115553 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553 2023-03-28 20:55:13,677 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553, entries=150, sequenceid=37, filesize=11.6 K 2023-03-28 20:55:13,677 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c39ece258f6d4e2ba19093b8d8052b5f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f 2023-03-28 20:55:13,681 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f, entries=150, sequenceid=37, filesize=11.6 K 2023-03-28 20:55:13,682 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 1d7220861206a2835f715e6061645bc2 in 500ms, sequenceid=37, compaction requested=false 2023-03-28 20:55:13,682 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:13,682 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:13,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:13,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:13,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d7ae11[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:13,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:13,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:13,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:13,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:13,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:13,695 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:13,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:13,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:13,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:13,695 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:13,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:13,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,708 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,708 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:13,708 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:13,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,809 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:13,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:13,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:13,812 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:13,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:13,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:13,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:13,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:13,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:13,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:13,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:13,816 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:13,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:13,819 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d6254[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:13,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:13,821 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:13,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:13,824 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:13,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,824 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:13,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:13,826 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:13,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:13,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:13,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:13,827 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:13,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:13,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,834 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:13,834 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:13,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:13,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,934 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:13,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:13,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:13,937 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:13,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:13,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:13,938 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,938 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:13,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:13,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:13,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:13,941 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:13,943 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ee7e41[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:13,945 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:13,945 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,946 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:13,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:13,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:13,952 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:13,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:13,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:13,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:13,952 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:13,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:13,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:13,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:13,959 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:13,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:13,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:13,959 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,959 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:13,959 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:13,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:13,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:13,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:14,060 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:14,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:14,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:14,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:14,063 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:14,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:14,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:14,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:14,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:14,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,065 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:14,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:14,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:14,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:14,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:14,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:14,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:14,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:14,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:14,067 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-28 20:55:14,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:14,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:14,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:14,068 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:14,068 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:14,068 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:14,081 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328765b2c1a1efb477d8aafcf35d831c604 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328765b2c1a1efb477d8aafcf35d831c604 2023-03-28 20:55:14,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:14,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036914124, max timestamp=1680036914132Num uniques 859; 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036914124, max timestamp=1680036914132Num uniques 859; 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036914124, max timestamp=1680036914132Num uniques 859; 2023-03-28 20:55:14,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:14,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036914151, max timestamp=1680036914160Num uniques 1018; 2023-03-28 20:55:14,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036914151, max timestamp=1680036914160Num uniques 1018; 2023-03-28 20:55:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036974154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:14,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036914151, max timestamp=1680036914160Num uniques 1018; 2023-03-28 20:55:14,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:14,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680036914177, max timestamp=1680036914184Num uniques 1027; 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680036914177, max timestamp=1680036914184Num uniques 1027; 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680036914177, max timestamp=1680036914184Num uniques 1027; 2023-03-28 20:55:14,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-28 20:55:14,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680036914215, max timestamp=1680036914230Num uniques 1136; 2023-03-28 20:55:14,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-03-28 20:55:14,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036974229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680036914215, max timestamp=1680036914230Num uniques 1136; 2023-03-28 20:55:14,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-03-28 20:55:14,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680036914215, max timestamp=1680036914230Num uniques 1136; 2023-03-28 20:55:14,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-03-28 20:55:14,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036974260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036974260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036974263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036974264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036974333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036974365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036974367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036974373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036974474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,494 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=46, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/4f4a7d19471a4b368401bca287531e53 2023-03-28 20:55:14,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036974537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036974571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036974573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036974577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,618 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:55:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036974780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036974843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036974877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036974878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036974882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,902 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/8bc32c7630a44aec8d0c148f0e2db22f 2023-03-28 20:55:14,913 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/d375f10aa55947e593bdb2d44101a040 2023-03-28 20:55:14,918 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/4f4a7d19471a4b368401bca287531e53 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53 2023-03-28 20:55:14,922 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53, entries=200, sequenceid=46, filesize=31.9 K 2023-03-28 20:55:14,923 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/8bc32c7630a44aec8d0c148f0e2db22f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f 2023-03-28 20:55:14,926 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f, entries=100, sequenceid=46, filesize=9.3 K 2023-03-28 20:55:14,927 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/d375f10aa55947e593bdb2d44101a040 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040 2023-03-28 20:55:14,931 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040, entries=100, sequenceid=46, filesize=9.3 K 2023-03-28 20:55:14,932 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 1d7220861206a2835f715e6061645bc2 in 865ms, sequenceid=46, compaction requested=true 2023-03-28 20:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:14,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:14,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,935 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:14,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c528d9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:14,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:14,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:14,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:14,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:14,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:14,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:14,946 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:14,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:14,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:14,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:14,946 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:14,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:14,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,948 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:14,948 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:14,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:14,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:14,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:14,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:14,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:14,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:14,955 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:14,955 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:14,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:14,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:15,055 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:15,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:15,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:15,058 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:15,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:15,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:15,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:15,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:15,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:15,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:15,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:15,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,061 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:15,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:15,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:15,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:15,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:15,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:15,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:15,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:15,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:15,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:15,062 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:15,062 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:15,062 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=161.02 KB heapSize=443.39 KB 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:15,063 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:15,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:15,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:15,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036975304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036975353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036975383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036975386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036975388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036975406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,475 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328d0321eee7c81462dae87ca4296bbaff2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d0321eee7c81462dae87ca4296bbaff2 2023-03-28 20:55:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036975612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:15,880 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e4ecc313a9f94fed9f4e9afe7c9acb64 2023-03-28 20:55:15,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036975917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,299 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/d2060068781c47e69ef1a20d50b16cc4 2023-03-28 20:55:16,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:16,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036976364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:16,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036976391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:16,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036976393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:16,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036976403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:16,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036976427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,710 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/6c216bfb42444f9a867df8dff3644f2f 2023-03-28 20:55:16,714 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e4ecc313a9f94fed9f4e9afe7c9acb64 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64 2023-03-28 20:55:16,717 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64, entries=150, sequenceid=134, filesize=25.3 K 2023-03-28 20:55:16,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/d2060068781c47e69ef1a20d50b16cc4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4 2023-03-28 20:55:16,721 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4, entries=150, sequenceid=134, filesize=11.8 K 2023-03-28 20:55:16,722 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/6c216bfb42444f9a867df8dff3644f2f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f 2023-03-28 20:55:16,725 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f, entries=150, sequenceid=134, filesize=11.8 K 2023-03-28 20:55:16,726 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~443.34 KB/453984, currentSize=26.84 KB/27480 for 1d7220861206a2835f715e6061645bc2 in 1664ms, sequenceid=134, compaction requested=true 2023-03-28 20:55:16,726 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:16,726 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:16,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:16,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:16,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:16,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa8b1f6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:16,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:16,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:16,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:16,734 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:16,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:16,737 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:16,737 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:16,737 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:16,737 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:16,737 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:16,739 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,739 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,739 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:16,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,741 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:16,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:16,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:16,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:16,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,743 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:16,744 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:16,744 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,744 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:16,744 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:16,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:16,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:16,844 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:16,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:16,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:16,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:16,849 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:16,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:16,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:16,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:16,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:16,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:16,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:16,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:16,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:16,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:16,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:16,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:16,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:16,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:16,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:16,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:16,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:16,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:16,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:16,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:16,853 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:16,870 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303286d62597acf364e9d9b77188789e45d78 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286d62597acf364e9d9b77188789e45d78 2023-03-28 20:55:17,275 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=141, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/c91fa6a8d1e541ce93769a72c8d26348 2023-03-28 20:55:17,286 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/aa39f3eed79840c4900dbaa2651359ed 2023-03-28 20:55:17,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:17,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:17,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917544, max timestamp=1680036917566Num uniques 859; 2023-03-28 20:55:17,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917544, max timestamp=1680036917566Num uniques 859; 2023-03-28 20:55:17,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917544, max timestamp=1680036917566Num uniques 859; 2023-03-28 20:55:17,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036917646, max timestamp=1680036917673Num uniques 1018; 2023-03-28 20:55:17,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:17,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036917646, max timestamp=1680036917673Num uniques 1018; 2023-03-28 20:55:17,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036917646, max timestamp=1680036917673Num uniques 1018; 2023-03-28 20:55:17,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:17,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:17,695 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/62968c1358f94096873beeeb6ee0e447 2023-03-28 20:55:17,698 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/c91fa6a8d1e541ce93769a72c8d26348 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348 2023-03-28 20:55:17,702 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348, entries=100, sequenceid=141, filesize=18.5 K 2023-03-28 20:55:17,702 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/aa39f3eed79840c4900dbaa2651359ed as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed 2023-03-28 20:55:17,706 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed, entries=100, sequenceid=141, filesize=9.4 K 2023-03-28 20:55:17,706 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/62968c1358f94096873beeeb6ee0e447 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447 2023-03-28 20:55:17,709 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447, entries=100, sequenceid=141, filesize=9.4 K 2023-03-28 20:55:17,710 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for 1d7220861206a2835f715e6061645bc2 in 857ms, sequenceid=141, compaction requested=true 2023-03-28 20:55:17,710 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:17,710 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:17,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:17,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=60.38 KB heapSize=165 KB 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:17,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:17,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,713 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a574d35[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:17,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:17,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:17,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:17,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:17,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:17,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:17,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:17,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:17,723 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:17,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:17,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:17,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:17,723 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:17,723 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328e4c82f8c450d4594830e63e75b07bfe4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328e4c82f8c450d4594830e63e75b07bfe4 2023-03-28 20:55:17,725 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,725 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,725 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,725 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:17,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:17,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:17,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:17,731 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:17,730 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,731 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:17,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:17,730 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=184, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/cd074809b3924c7781d55ee1f4b9a026 2023-03-28 20:55:17,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/75413c61d2bb4213aa3af8f0d907b80a 2023-03-28 20:55:17,831 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:17,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:17,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:17,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:17,835 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:17,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:17,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:17,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:17,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:17,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:17,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:17,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:17,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:17,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:17,841 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:17,841 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:17,841 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:17,841 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,843 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:17,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2d3e48[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:17,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:17,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:17,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:17,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:17,846 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,846 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:17,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:17,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:17,851 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:17,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:17,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:17,851 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:17,851 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:17,853 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,857 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:17,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:17,859 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:17,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:17,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:17,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,860 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:17,860 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:17,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:17,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917857, max timestamp=1680036917900Num uniques 859; 2023-03-28 20:55:17,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917857, max timestamp=1680036917900Num uniques 859; 2023-03-28 20:55:17,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036917857, max timestamp=1680036917900Num uniques 859; 2023-03-28 20:55:17,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:17,960 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:17,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:17,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:17,963 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:17,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:17,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:17,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:17,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:17,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:17,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:17,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:17,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:17,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:17,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:17,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:17,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:17,973 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454748f4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:17,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:17,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:17,977 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,977 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,978 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:17,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:17,981 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:17,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:17,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:17,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:17,981 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:17,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:17,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:17,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:17,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:17,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:17,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:17,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:17,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:17,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:17,990 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:17,990 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:18,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036978046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,090 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:18,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:18,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:18,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:18,093 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,095 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:18,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:18,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:18,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:18,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:18,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:18,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:18,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:18,097 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:18,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:18,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:18,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:18,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2f66ab[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:18,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:18,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,102 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:18,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:18,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:18,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:18,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:18,105 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:18,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:18,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:18,107 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:18,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:18,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:18,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:18,107 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:18,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:18,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:18,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:18,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,114 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:18,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:18,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:18,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,114 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:18,114 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:18,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:18,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680036978155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/7f6efb91da9148e3ba59033f7368cf48 2023-03-28 20:55:18,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/cd074809b3924c7781d55ee1f4b9a026 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026 2023-03-28 20:55:18,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026, entries=200, sequenceid=184, filesize=32.1 K 2023-03-28 20:55:18,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/75413c61d2bb4213aa3af8f0d907b80a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a 2023-03-28 20:55:18,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a, entries=150, sequenceid=184, filesize=11.8 K 2023-03-28 20:55:18,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/7f6efb91da9148e3ba59033f7368cf48 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48 2023-03-28 20:55:18,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48, entries=150, sequenceid=184, filesize=11.8 K 2023-03-28 20:55:18,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.95 KB/168912, currentSize=127.47 KB/130530 for 1d7220861206a2835f715e6061645bc2 in 486ms, sequenceid=184, compaction requested=true 2023-03-28 20:55:18,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:18,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:18,197 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:55:18,197 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:55:18,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:18,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:18,199 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168794 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:55:18,199 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/A is initiating minor compaction (all files) 2023-03-28 20:55:18,199 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/A in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,199 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=164.8 K 2023-03-28 20:55:18,199 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67028 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:55:18,199 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/B is initiating minor compaction (all files) 2023-03-28 20:55:18,199 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ff500f6f854040c4b6159f15656ec5fd, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036911869 2023-03-28 20:55:18,199 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/B in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,200 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=65.5 K 2023-03-28 20:55:18,200 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ea98ca487a4a48f1b2fa40c45747deb8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036911895 2023-03-28 20:55:18,200 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 497afdf67a8e47eb8a8325fc6b2749b8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036911869 2023-03-28 20:55:18,201 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4f4a7d19471a4b368401bca287531e53, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680036914054 2023-03-28 20:55:18,201 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4ed6912c8bef429ebdf1126f82115553, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036911895 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e4ecc313a9f94fed9f4e9afe7c9acb64, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680036914259 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 8bc32c7630a44aec8d0c148f0e2db22f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680036914056 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c91fa6a8d1e541ce93769a72c8d26348, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680036915298 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d2060068781c47e69ef1a20d50b16cc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680036914259 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cd074809b3924c7781d55ee1f4b9a026, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:18,202 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting aa39f3eed79840c4900dbaa2651359ed, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680036915298 2023-03-28 20:55:18,203 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 75413c61d2bb4213aa3af8f0d907b80a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:18,213 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#A#compaction#385 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:18,213 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#B#compaction#386 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:18,215 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:18,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:18,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:18,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:18,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:18,230 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:18,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:18,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:18,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:18,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,232 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:18,232 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:18,232 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,232 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:18,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:18,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:18,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:18,234 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:18,234 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:18,234 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:18,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:18,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:18,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036978399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036978401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036978402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036978412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,421 DEBUG [Thread-2682] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:18,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036978427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,431 DEBUG [Thread-2678] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:18,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036978508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036978511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036978511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,634 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/25776ff8967444fdb00df4d7ed330910 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/25776ff8967444fdb00df4d7ed330910 2023-03-28 20:55:18,638 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1d7220861206a2835f715e6061645bc2/B of 1d7220861206a2835f715e6061645bc2 into 25776ff8967444fdb00df4d7ed330910(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:18,639 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:18,639 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/B, priority=10, startTime=1680036918197; duration=0sec 2023-03-28 20:55:18,639 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:18,639 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:55:18,640 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67028 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:55:18,640 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/C is initiating minor compaction (all files) 2023-03-28 20:55:18,640 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/C in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:18,640 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=65.5 K 2023-03-28 20:55:18,641 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bda98b77a3e64ea19ad21bd7ae9dab96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036911869 2023-03-28 20:55:18,641 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c39ece258f6d4e2ba19093b8d8052b5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680036911895 2023-03-28 20:55:18,641 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d375f10aa55947e593bdb2d44101a040, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680036914056 2023-03-28 20:55:18,642 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6c216bfb42444f9a867df8dff3644f2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680036914259 2023-03-28 20:55:18,642 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 62968c1358f94096873beeeb6ee0e447, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680036915298 2023-03-28 20:55:18,642 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7f6efb91da9148e3ba59033f7368cf48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:18,648 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303288c30aeb22a4f46c5b0e938df390ff330 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288c30aeb22a4f46c5b0e938df390ff330 2023-03-28 20:55:18,652 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/a01578119fb04edd84ed18aae6c46381 2023-03-28 20:55:18,655 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#C#compaction#388 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:18,662 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/29ed5665db9040679aabaf753c3b79fe 2023-03-28 20:55:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036978716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036978718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:18,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:18,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036978720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036979024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036979028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036979028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,042 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/c56e8ac12ddf42cda5e49fe8b83e08f4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c56e8ac12ddf42cda5e49fe8b83e08f4 2023-03-28 20:55:19,046 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1d7220861206a2835f715e6061645bc2/A of 1d7220861206a2835f715e6061645bc2 into c56e8ac12ddf42cda5e49fe8b83e08f4(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:19,046 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:19,046 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/A, priority=10, startTime=1680036918197; duration=0sec 2023-03-28 20:55:19,046 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:19,063 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/ad1eb766eaef44ccbb831909abcfe870 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/ad1eb766eaef44ccbb831909abcfe870 2023-03-28 20:55:19,066 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1d7220861206a2835f715e6061645bc2/C of 1d7220861206a2835f715e6061645bc2 into ad1eb766eaef44ccbb831909abcfe870(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:19,066 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:19,066 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/C, priority=10, startTime=1680036918197; duration=0sec 2023-03-28 20:55:19,066 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:19,072 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/6c9a28cb7d8f4170a18c42d32afc51fa 2023-03-28 20:55:19,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/a01578119fb04edd84ed18aae6c46381 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381 2023-03-28 20:55:19,077 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381, entries=150, sequenceid=220, filesize=25.3 K 2023-03-28 20:55:19,078 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/29ed5665db9040679aabaf753c3b79fe as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe 2023-03-28 20:55:19,081 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe, entries=150, sequenceid=220, filesize=11.8 K 2023-03-28 20:55:19,082 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/6c9a28cb7d8f4170a18c42d32afc51fa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa 2023-03-28 20:55:19,084 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa, entries=150, sequenceid=220, filesize=11.8 K 2023-03-28 20:55:19,085 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1d7220861206a2835f715e6061645bc2 in 851ms, sequenceid=220, compaction requested=false 2023-03-28 20:55:19,085 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:19,085 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:19,088 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,088 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:19,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:19,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a61bc3e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,091 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:19,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:19,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:19,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:19,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:19,097 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:19,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:19,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:19,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:19,098 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:19,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,099 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:19,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:19,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:19,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:19,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:19,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:19,107 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:19,107 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:19,107 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:19,207 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:19,211 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:19,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:19,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:19,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:19,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:19,213 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:19,213 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:19,213 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,213 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:19,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:19,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:19,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:19,214 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:19,215 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:19,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:19,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036979567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036919560, max timestamp=1680036919572Num uniques 859; 2023-03-28 20:55:19,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:19,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036919560, max timestamp=1680036919572Num uniques 859; 2023-03-28 20:55:19,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:19,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036919560, max timestamp=1680036919572Num uniques 859; 2023-03-28 20:55:19,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:19,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036979617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036979618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,628 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032863028a08b3d4442e8af828b850141352 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032863028a08b3d4442e8af828b850141352 2023-03-28 20:55:19,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036979674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036979726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036979726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036979884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036979929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:19,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036979929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,033 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=234, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/48fe294173554e57abb8f5ef2a435043 2023-03-28 20:55:20,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036980189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036980232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036980233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/1f6c128876bc441c9542810bf2325d8e 2023-03-28 20:55:20,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036980696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036980737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:20,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036980740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,852 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/322f7354adc04f2eaa0f3925d3b8cb09 2023-03-28 20:55:20,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/48fe294173554e57abb8f5ef2a435043 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043 2023-03-28 20:55:20,859 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043, entries=150, sequenceid=234, filesize=25.3 K 2023-03-28 20:55:20,860 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/1f6c128876bc441c9542810bf2325d8e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e 2023-03-28 20:55:20,863 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e, entries=150, sequenceid=234, filesize=11.8 K 2023-03-28 20:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/322f7354adc04f2eaa0f3925d3b8cb09 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09 2023-03-28 20:55:20,867 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09, entries=150, sequenceid=234, filesize=11.8 K 2023-03-28 20:55:20,867 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 1d7220861206a2835f715e6061645bc2 in 1652ms, sequenceid=234, compaction requested=true 2023-03-28 20:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:20,867 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:20,868 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:20,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:20,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:20,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7b1244[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:20,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:20,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,876 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:20,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:20,877 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:20,880 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:20,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:20,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:20,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:20,881 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:20,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:20,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:20,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:20,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:20,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:20,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:20,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:20,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:20,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:20,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:20,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:20,887 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:20,887 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:20,987 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:20,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:20,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:20,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:20,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:20,991 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:20,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:20,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:20,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:20,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,992 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:20,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:20,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:20,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:20,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:20,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:20,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:20,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:20,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:20,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:20,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:20,996 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:20,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:21,008 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032893fdd381ff214615b620a964414afe91 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893fdd381ff214615b620a964414afe91 2023-03-28 20:55:21,417 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/330fb402024c48ffb83c34b8ad832a2a 2023-03-28 20:55:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:21,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:21,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:21,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036981742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:21,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036981745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:21,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036981754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,826 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/c03283d5cabe496e92916d1f9fa77994 2023-03-28 20:55:21,836 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c2f32914ba5e401d820f87c7871f33ea 2023-03-28 20:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/330fb402024c48ffb83c34b8ad832a2a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a 2023-03-28 20:55:21,842 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a, entries=150, sequenceid=271, filesize=25.4 K 2023-03-28 20:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/c03283d5cabe496e92916d1f9fa77994 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994 2023-03-28 20:55:21,846 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994, entries=150, sequenceid=271, filesize=11.9 K 2023-03-28 20:55:21,846 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c2f32914ba5e401d820f87c7871f33ea as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea 2023-03-28 20:55:21,849 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea, entries=150, sequenceid=271, filesize=11.9 K 2023-03-28 20:55:21,850 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1d7220861206a2835f715e6061645bc2 in 853ms, sequenceid=271, compaction requested=true 2023-03-28 20:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:21,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4004edb7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:21,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:21,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:21,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:21,855 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:21,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:21,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:21,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:21,858 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:21,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:21,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:21,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:21,859 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:21,859 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:21,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:21,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:21,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,865 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:21,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:21,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:21,866 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,866 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:21,866 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,866 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,966 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:21,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:21,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:21,969 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:21,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:21,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:21,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:21,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:21,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036921950, max timestamp=1680036921969Num uniques 859; 2023-03-28 20:55:21,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:21,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:21,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:21,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036921950, max timestamp=1680036921969Num uniques 859; 2023-03-28 20:55:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036921950, max timestamp=1680036921969Num uniques 859; 2023-03-28 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:21,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:21,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:21,973 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:21,973 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:21,973 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:21,973 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:21,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316fd122[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:21,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:21,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,984 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:21,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:21,994 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:21,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:21,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:21,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:21,994 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:21,996 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,996 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:21,996 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:21,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:21,998 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:21,999 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,002 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,002 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,003 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,003 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,003 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,103 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,107 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,109 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,111 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036982104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46980b7d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,120 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,121 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,132 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,134 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,135 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,135 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,139 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,141 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,141 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,142 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,142 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036982214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,242 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,245 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,246 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,246 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,248 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,249 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,250 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,250 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,250 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,250 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,251 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2116c8d8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,254 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,254 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,254 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,254 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,258 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,259 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,259 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,261 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,265 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,265 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,265 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,269 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032875a4514367f243c894c63c85b2abc77b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875a4514367f243c894c63c85b2abc77b 2023-03-28 20:55:22,284 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/4add1a10232745bc81fd68b201baafe9 2023-03-28 20:55:22,365 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,370 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,372 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,372 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,373 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,378 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4698e018[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,383 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,385 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,385 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,388 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,391 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,391 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,391 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036982416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680036982430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,437 DEBUG [Thread-2682] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:22,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48274 deadline: 1680036982437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,444 DEBUG [Thread-2678] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:22,491 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,495 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,498 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,500 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,500 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,500 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,500 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,502 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b1cf28[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,504 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,504 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,505 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,507 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,509 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,509 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,509 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,515 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,519 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,519 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,519 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,619 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,623 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,629 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,629 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,629 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,629 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f74fe80[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,630 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,630 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,633 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,635 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,635 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,636 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,636 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,642 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,642 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/644ef1cfed4e4db2bf71924e896259b1 2023-03-28 20:55:22,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:22,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036982725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,742 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,745 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,750 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,750 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,750 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,750 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,752 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1833b7be[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,754 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,754 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,755 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,761 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,761 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,764 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,767 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,767 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,767 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,868 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:22,871 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:22,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:22,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:22,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,873 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,873 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:22,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:22,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:22,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:22,875 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,875 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:22,875 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:22,875 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,878 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:22,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192a1c80[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:22,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:22,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:22,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:22,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:22,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:22,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:22,885 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:22,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:22,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:22,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:22,885 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:22,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:22,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:22,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:22,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:22,896 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:22,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:22,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:22,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:22,896 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:22,896 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:22,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:22,996 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:22,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:23,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:23,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:23,000 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:23,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:23,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:23,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,000 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,002 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,002 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,002 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,002 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:23,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:23,003 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,004 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:23,004 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:23,004 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:23,005 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,005 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ee55802[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:23,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:23,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,010 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:23,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:23,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:23,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:23,014 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:23,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:23,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:23,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:23,014 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:23,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,017 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:23,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:23,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,020 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:23,020 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:23,020 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a931b4b2b3494a1b832be6ee3abc81e8 2023-03-28 20:55:23,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/4add1a10232745bc81fd68b201baafe9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9 2023-03-28 20:55:23,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9, entries=200, sequenceid=282, filesize=32.3 K 2023-03-28 20:55:23,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/644ef1cfed4e4db2bf71924e896259b1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1 2023-03-28 20:55:23,120 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:23,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1, entries=150, sequenceid=282, filesize=11.9 K 2023-03-28 20:55:23,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a931b4b2b3494a1b832be6ee3abc81e8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8 2023-03-28 20:55:23,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:23,123 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:23,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:23,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:23,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8, entries=150, sequenceid=282, filesize=11.9 K 2023-03-28 20:55:23,125 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1d7220861206a2835f715e6061645bc2 in 1267ms, sequenceid=282, compaction requested=true 2023-03-28 20:55:23,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:23,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:23,126 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:55:23,126 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,126 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:55:23,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:23,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:23,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:23,127 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137026 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:23,127 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/A is initiating minor compaction (all files) 2023-03-28 20:55:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,127 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:55:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:23,128 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/A in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,128 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/B is initiating minor compaction (all files) 2023-03-28 20:55:23,128 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c56e8ac12ddf42cda5e49fe8b83e08f4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=133.8 K 2023-03-28 20:55:23,128 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/B in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:23,128 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/25776ff8967444fdb00df4d7ed330910, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=59.3 K 2023-03-28 20:55:23,128 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,128 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:23,128 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c56e8ac12ddf42cda5e49fe8b83e08f4, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:23,128 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-28 20:55:23,128 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 25776ff8967444fdb00df4d7ed330910, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:23,128 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:23,128 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a01578119fb04edd84ed18aae6c46381, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680036918032 2023-03-28 20:55:23,128 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:23,129 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 29ed5665db9040679aabaf753c3b79fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680036918032 2023-03-28 20:55:23,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:23,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:23,129 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 48fe294173554e57abb8f5ef2a435043, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680036918398 2023-03-28 20:55:23,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:23,129 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1f6c128876bc441c9542810bf2325d8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680036918398 2023-03-28 20:55:23,129 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:23,129 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 330fb402024c48ffb83c34b8ad832a2a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680036919603 2023-03-28 20:55:23,129 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c03283d5cabe496e92916d1f9fa77994, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680036919603 2023-03-28 20:55:23,130 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4add1a10232745bc81fd68b201baafe9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921728 2023-03-28 20:55:23,130 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 644ef1cfed4e4db2bf71924e896259b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921735 2023-03-28 20:55:23,151 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#B#compaction#401 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:23,155 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#A#compaction#402 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:23,162 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032825fd7d5ded7d4ba6badafcfa8f425ef3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032825fd7d5ded7d4ba6badafcfa8f425ef3 2023-03-28 20:55:23,174 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/0fa405ba0a1d4a888a33e0de8932bc30 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/0fa405ba0a1d4a888a33e0de8932bc30 2023-03-28 20:55:23,176 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/5de2f0f4508f4f03bd0d922f81e05818 2023-03-28 20:55:23,178 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1d7220861206a2835f715e6061645bc2/B of 1d7220861206a2835f715e6061645bc2 into 0fa405ba0a1d4a888a33e0de8932bc30(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:23,178 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:23,178 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/B, priority=11, startTime=1680036923126; duration=0sec 2023-03-28 20:55:23,178 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:23,178 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:55:23,179 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:55:23,180 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/C is initiating minor compaction (all files) 2023-03-28 20:55:23,180 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/C in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,180 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/ad1eb766eaef44ccbb831909abcfe870, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=59.3 K 2023-03-28 20:55:23,180 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ad1eb766eaef44ccbb831909abcfe870, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680036917680 2023-03-28 20:55:23,180 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6c9a28cb7d8f4170a18c42d32afc51fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680036918032 2023-03-28 20:55:23,181 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 322f7354adc04f2eaa0f3925d3b8cb09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680036918398 2023-03-28 20:55:23,181 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c2f32914ba5e401d820f87c7871f33ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680036919603 2023-03-28 20:55:23,182 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a931b4b2b3494a1b832be6ee3abc81e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921735 2023-03-28 20:55:23,189 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#C#compaction#404 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:23,204 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/b9b78f08366e4d4c818df9952e3bdfb6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b9b78f08366e4d4c818df9952e3bdfb6 2023-03-28 20:55:23,207 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1d7220861206a2835f715e6061645bc2/C of 1d7220861206a2835f715e6061645bc2 into b9b78f08366e4d4c818df9952e3bdfb6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:23,207 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:23,207 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/C, priority=11, startTime=1680036923126; duration=0sec 2023-03-28 20:55:23,207 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:23,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:23,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:23,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036983277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036983384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036983589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,599 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/a0fddeb37cc3406c8a40063e56b0a26e 2023-03-28 20:55:23,606 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/aa032192b498430fa4e9cda1dd8e2fc6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/aa032192b498430fa4e9cda1dd8e2fc6 2023-03-28 20:55:23,614 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1d7220861206a2835f715e6061645bc2/A of 1d7220861206a2835f715e6061645bc2 into aa032192b498430fa4e9cda1dd8e2fc6(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:23,614 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:23,614 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/A, priority=11, startTime=1680036923126; duration=0sec 2023-03-28 20:55:23,614 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:23,623 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/81466c8f96cb4ce1bd1818149983e08a 2023-03-28 20:55:23,627 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/5de2f0f4508f4f03bd0d922f81e05818 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818 2023-03-28 20:55:23,630 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818, entries=150, sequenceid=318, filesize=25.4 K 2023-03-28 20:55:23,630 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/a0fddeb37cc3406c8a40063e56b0a26e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e 2023-03-28 20:55:23,633 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e, entries=150, sequenceid=318, filesize=11.9 K 2023-03-28 20:55:23,634 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/81466c8f96cb4ce1bd1818149983e08a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a 2023-03-28 20:55:23,637 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a, entries=150, sequenceid=318, filesize=11.9 K 2023-03-28 20:55:23,637 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1d7220861206a2835f715e6061645bc2 in 509ms, sequenceid=318, compaction requested=false 2023-03-28 20:55:23,637 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:23,637 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:23,638 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:23,638 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,638 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:23,638 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:23,639 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,639 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@364d25a0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:23,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:23,641 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,642 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:23,642 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,642 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:23,644 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:23,646 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:23,646 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:23,646 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:23,646 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:23,646 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:23,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,649 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:23,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:23,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,653 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:23,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,653 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:23,653 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:23,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:23,753 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:23,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:23,760 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:23,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:23,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:23,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,762 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:23,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:23,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:23,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,764 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:23,765 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:23,765 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:23,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,769 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328ccc4516d688549cd8858a0f957bca0c9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ccc4516d688549cd8858a0f957bca0c9 2023-03-28 20:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c6ea90[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:23,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:23,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,772 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:23,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:23,775 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:23,775 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,775 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,775 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,776 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=333, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e45f47d492494264a477a9d54f17d3eb 2023-03-28 20:55:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:23,778 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:23,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:23,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:23,778 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:23,778 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:23,782 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,782 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,782 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/3cfcec97330340ae913a8f26e077c2f7 2023-03-28 20:55:23,791 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,792 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:23,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:23,795 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:23,795 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:23,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036983803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036983807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,896 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:23,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:23,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:23,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036983897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:23,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:23,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,900 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,900 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:23,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:23,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:23,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:23,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:23,902 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,904 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a413e74[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:23,906 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:23,907 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:23,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:23,910 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:23,913 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:23,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:23,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:23,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:23,913 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:23,914 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:23,914 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,915 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036983910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:23,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:23,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036983912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:23,919 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:23,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:23,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:23,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:23,919 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:23,919 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:23,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,920 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:23,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:23,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,020 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:24,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:24,023 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:24,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:24,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:24,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:24,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:24,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:24,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:24,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:24,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:24,028 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:24,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:24,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e505c76[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:24,031 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:24,031 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,032 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:24,033 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,033 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:24,034 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:24,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:24,036 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:24,036 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:24,039 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,039 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,039 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:24,040 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:24,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:24,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,044 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:24,044 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:24,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:24,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:24,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036984117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036984118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,144 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:24,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:24,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:24,147 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:24,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:24,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:24,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,152 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:24,156 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:24,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:24,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:24,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:24,162 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,162 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:24,162 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:24,162 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,164 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:24,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116f1039[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:24,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:24,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,169 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:24,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:24,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:24,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:24,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:24,173 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:24,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:24,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:24,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:24,174 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:24,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:24,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,176 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:24,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:24,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,177 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:24,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:24,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:24,183 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:24,183 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:24,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:24,183 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:24,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/75da501de6ea4f69bf46a0687fa40844 2023-03-28 20:55:24,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e45f47d492494264a477a9d54f17d3eb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb 2023-03-28 20:55:24,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb, entries=200, sequenceid=333, filesize=32.3 K 2023-03-28 20:55:24,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/3cfcec97330340ae913a8f26e077c2f7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7 2023-03-28 20:55:24,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7, entries=150, sequenceid=333, filesize=11.9 K 2023-03-28 20:55:24,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/75da501de6ea4f69bf46a0687fa40844 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844 2023-03-28 20:55:24,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844, entries=150, sequenceid=333, filesize=11.9 K 2023-03-28 20:55:24,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1d7220861206a2835f715e6061645bc2 in 464ms, sequenceid=333, compaction requested=true 2023-03-28 20:55:24,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:24,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:24,216 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:24,217 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:24,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:24,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:24,217 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:24,217 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:24,217 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/A is initiating minor compaction (all files) 2023-03-28 20:55:24,217 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/B is initiating minor compaction (all files) 2023-03-28 20:55:24,218 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/A in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,218 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/B in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,218 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/aa032192b498430fa4e9cda1dd8e2fc6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=83.5 K 2023-03-28 20:55:24,218 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/0fa405ba0a1d4a888a33e0de8932bc30, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=36.1 K 2023-03-28 20:55:24,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting aa032192b498430fa4e9cda1dd8e2fc6, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921735 2023-03-28 20:55:24,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0fa405ba0a1d4a888a33e0de8932bc30, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921735 2023-03-28 20:55:24,218 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5de2f0f4508f4f03bd0d922f81e05818, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680036921990 2023-03-28 20:55:24,218 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a0fddeb37cc3406c8a40063e56b0a26e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680036921990 2023-03-28 20:55:24,219 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e45f47d492494264a477a9d54f17d3eb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:24,220 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3cfcec97330340ae913a8f26e077c2f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:24,228 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#B#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:24,233 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#A#compaction#410 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:24,283 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:24,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:24,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:24,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:24,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:24,287 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:24,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:24,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:24,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,289 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:24,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:24,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:24,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:24,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:24,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:24,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:24,291 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:55:24,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:24,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:24,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:24,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:24,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036984429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036984432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036984434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036984535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036984536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036984539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,641 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/6410406308224466b171891eb65c35f9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6410406308224466b171891eb65c35f9 2023-03-28 20:55:24,645 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/B of 1d7220861206a2835f715e6061645bc2 into 6410406308224466b171891eb65c35f9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:24,645 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:24,645 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/B, priority=13, startTime=1680036924216; duration=0sec 2023-03-28 20:55:24,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:24,646 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:24,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:24,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/C is initiating minor compaction (all files) 2023-03-28 20:55:24,647 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/C in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:24,647 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b9b78f08366e4d4c818df9952e3bdfb6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=36.1 K 2023-03-28 20:55:24,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b9b78f08366e4d4c818df9952e3bdfb6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680036921735 2023-03-28 20:55:24,647 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 81466c8f96cb4ce1bd1818149983e08a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680036921990 2023-03-28 20:55:24,648 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 75da501de6ea4f69bf46a0687fa40844, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:24,654 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#C#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:24,663 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/cae0d82a19044a5e8cc1f896b36db774 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/cae0d82a19044a5e8cc1f896b36db774 2023-03-28 20:55:24,666 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/C of 1d7220861206a2835f715e6061645bc2 into cae0d82a19044a5e8cc1f896b36db774(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:24,666 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:24,666 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/C, priority=13, startTime=1680036924217; duration=0sec 2023-03-28 20:55:24,666 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:24,709 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303284c2adb1f17f64678bf9b2ffd6c25f60f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284c2adb1f17f64678bf9b2ffd6c25f60f 2023-03-28 20:55:24,713 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=353, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/cb4544b66a084f9198353d8357926bfc 2023-03-28 20:55:24,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036984741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036984741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:24,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:24,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036984743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036985044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036985044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036985047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,058 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/ae3ed6a30da84613aee8feb23b94b09b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ae3ed6a30da84613aee8feb23b94b09b 2023-03-28 20:55:25,062 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/A of 1d7220861206a2835f715e6061645bc2 into ae3ed6a30da84613aee8feb23b94b09b(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:25,062 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:25,062 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/A, priority=13, startTime=1680036924216; duration=0sec 2023-03-28 20:55:25,062 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:25,122 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/3839dc12ae244fd1b8132679f6906613 2023-03-28 20:55:25,531 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/9b52d96af3094c23875747e2de586b69 2023-03-28 20:55:25,535 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/cb4544b66a084f9198353d8357926bfc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc 2023-03-28 20:55:25,538 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc, entries=150, sequenceid=353, filesize=25.4 K 2023-03-28 20:55:25,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/3839dc12ae244fd1b8132679f6906613 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613 2023-03-28 20:55:25,542 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613, entries=150, sequenceid=353, filesize=11.9 K 2023-03-28 20:55:25,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/9b52d96af3094c23875747e2de586b69 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69 2023-03-28 20:55:25,545 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69, entries=150, sequenceid=353, filesize=11.9 K 2023-03-28 20:55:25,546 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1d7220861206a2835f715e6061645bc2 in 1255ms, sequenceid=353, compaction requested=false 2023-03-28 20:55:25,546 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:25,546 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:25,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:55:25,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:25,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:25,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:25,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:25,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:25,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f9eaee[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:25,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:25,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,559 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,559 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:25,559 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:25,559 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:25,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,559 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:25,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:25,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:25,564 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:25,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:25,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:25,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:25,564 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:25,566 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,566 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,566 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,566 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,568 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:25,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,572 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:25,572 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036985587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036985588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036985592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,672 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:25,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:25,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:25,675 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:25,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:25,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:25,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,677 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:25,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:25,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:25,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:25,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:25,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:25,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:25,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,681 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6638c4bb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:25,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:25,686 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,687 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:25,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:25,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:25,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:25,692 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:25,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:25,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:25,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:25,693 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:25,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,694 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036985693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036985693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:25,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:25,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,701 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:25,701 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:25,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:25,700 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036985699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,801 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:25,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:25,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:25,804 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:25,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:25,805 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:25,805 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,805 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,806 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:25,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:25,809 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,809 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:25,809 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:25,809 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:25,811 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,811 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:25,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4797e6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:25,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:25,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:25,815 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:25,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,815 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:25,819 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:25,819 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:25,819 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:25,819 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:25,819 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:25,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,822 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,828 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,829 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:25,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,829 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:25,829 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036985900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036985900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:25,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036985908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,929 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:25,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:25,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:25,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:25,940 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:25,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:25,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:25,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,942 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:25,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:25,944 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,944 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:25,944 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:25,944 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:25,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:25,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c0b44f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:25,950 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,951 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:25,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,952 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:25,958 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:25,960 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:25,960 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:25,960 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:25,960 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:25,960 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:25,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:25,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:25,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:25,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:25,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:25,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:25,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:25,967 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:25,967 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:25,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:25,968 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328cb76975b74c3482d91dc6d0e7786d645 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb76975b74c3482d91dc6d0e7786d645 2023-03-28 20:55:26,067 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,071 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,072 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,075 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,077 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,077 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b49843[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,086 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,086 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,087 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:26,093 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,093 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,193 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,197 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,198 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,200 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,200 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,200 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,200 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39099ad2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036986203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,208 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036986204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,212 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,212 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,212 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,212 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,212 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,214 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,214 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,214 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036986216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,219 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,219 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,220 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,220 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,320 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,323 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,323 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,323 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,323 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,325 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,325 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,325 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,326 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,327 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,327 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,327 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,327 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,330 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,331 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,331 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f67d2b2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,332 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,332 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,333 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,333 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,333 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,335 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,335 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,335 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,335 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,338 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,338 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,338 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,338 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,338 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,343 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,343 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,343 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,345 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,348 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,348 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:26,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,373 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/de37a3a6dc504db7ba558c1af6af79d0 2023-03-28 20:55:26,448 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,451 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,453 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,455 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,455 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,455 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,455 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,457 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c379dd[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,459 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,459 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,461 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,461 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,462 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,462 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,462 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,462 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,464 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,464 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,466 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,466 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,466 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,468 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,470 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,470 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,470 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,570 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,573 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,576 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,576 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,578 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,578 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,578 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,578 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,580 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54089226[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,582 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,582 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,583 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,584 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,594 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,594 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,595 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,595 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:26,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,695 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,698 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,698 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,699 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:26,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:26,703 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21224d7f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:26,705 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:26,705 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,707 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:26,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:26,707 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,707 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:26,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:26,709 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036986710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:26,712 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:26,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:26,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:26,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:26,712 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:26,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:26,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:26,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:26,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48194 deadline: 1680036986711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:26,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:26,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:26,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:26,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:26,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:26,723 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:26,723 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:26,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:26,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48210 deadline: 1680036986719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/60cd99591d6f48b2b1cfbbc130b845ee 2023-03-28 20:55:26,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/4f4fae27eb4240fbb41650c64006ec08 2023-03-28 20:55:26,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/de37a3a6dc504db7ba558c1af6af79d0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0 2023-03-28 20:55:26,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0, entries=250, sequenceid=370, filesize=39.2 K 2023-03-28 20:55:26,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/60cd99591d6f48b2b1cfbbc130b845ee as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee 2023-03-28 20:55:26,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee, entries=150, sequenceid=370, filesize=11.9 K 2023-03-28 20:55:26,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/4f4fae27eb4240fbb41650c64006ec08 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08 2023-03-28 20:55:26,823 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:26,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:26,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:26,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08, entries=150, sequenceid=370, filesize=11.9 K 2023-03-28 20:55:26,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1d7220861206a2835f715e6061645bc2 in 1271ms, sequenceid=370, compaction requested=true 2023-03-28 20:55:26,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:26,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:26,826 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:26,826 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:26,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:26,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:26,828 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92700 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:26,828 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/A is initiating minor compaction (all files) 2023-03-28 20:55:26,828 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/A in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,828 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ae3ed6a30da84613aee8feb23b94b09b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=90.5 K 2023-03-28 20:55:26,828 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:26,828 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/B is initiating minor compaction (all files) 2023-03-28 20:55:26,829 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/B in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,829 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6410406308224466b171891eb65c35f9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=36.2 K 2023-03-28 20:55:26,829 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ae3ed6a30da84613aee8feb23b94b09b, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:26,829 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cb4544b66a084f9198353d8357926bfc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680036923793 2023-03-28 20:55:26,831 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6410406308224466b171891eb65c35f9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:26,831 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3839dc12ae244fd1b8132679f6906613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680036923793 2023-03-28 20:55:26,832 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting de37a3a6dc504db7ba558c1af6af79d0, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680036924423 2023-03-28 20:55:26,832 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 60cd99591d6f48b2b1cfbbc130b845ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680036924423 2023-03-28 20:55:26,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:26,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:26,836 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:26,836 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,836 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:26,836 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:26,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:26,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:26,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:26,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:26,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:26,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:26,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:26,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:26,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:26,853 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:26,856 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:26,873 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#A#compaction#418 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:26,969 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303286333105e562c4c148f5265d15cafb194 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286333105e562c4c148f5265d15cafb194 2023-03-28 20:55:26,986 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=388, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/b47ee558e789490c88641d926792dbc5 2023-03-28 20:55:27,004 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/1838b65316654f29976fb5ea473729f6 2023-03-28 20:55:27,022 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/78060e479d83459b853d3646b0d4b1b7 2023-03-28 20:55:27,025 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/b47ee558e789490c88641d926792dbc5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5 2023-03-28 20:55:27,029 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5, entries=150, sequenceid=388, filesize=25.4 K 2023-03-28 20:55:27,030 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/1838b65316654f29976fb5ea473729f6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6 2023-03-28 20:55:27,035 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6, entries=150, sequenceid=388, filesize=11.9 K 2023-03-28 20:55:27,035 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/78060e479d83459b853d3646b0d4b1b7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7 2023-03-28 20:55:27,039 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7, entries=150, sequenceid=388, filesize=11.9 K 2023-03-28 20:55:27,039 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 1d7220861206a2835f715e6061645bc2 in 183ms, sequenceid=388, compaction requested=true 2023-03-28 20:55:27,039 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,039 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c1e9ab[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,045 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,048 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,050 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,051 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,053 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,053 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,053 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,054 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,057 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,057 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,058 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,158 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,161 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,161 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,161 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,161 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,166 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,166 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,166 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,166 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,168 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423e63ff[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,173 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,175 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,176 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,177 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,182 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,182 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,182 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,183 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,183 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,283 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,286 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,288 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b7503c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,298 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,302 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,310 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,310 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,325 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/6dd36f85a31743e3a01fa4459497054d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6dd36f85a31743e3a01fa4459497054d 2023-03-28 20:55:27,329 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/B of 1d7220861206a2835f715e6061645bc2 into 6dd36f85a31743e3a01fa4459497054d(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:27,329 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,329 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/B, priority=13, startTime=1680036926826; duration=0sec 2023-03-28 20:55:27,329 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:27,329 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:27,330 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:27,330 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/C is initiating minor compaction (all files) 2023-03-28 20:55:27,330 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/C in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,330 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/cae0d82a19044a5e8cc1f896b36db774, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=48.1 K 2023-03-28 20:55:27,331 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cae0d82a19044a5e8cc1f896b36db774, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680036923256 2023-03-28 20:55:27,331 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 9b52d96af3094c23875747e2de586b69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680036923793 2023-03-28 20:55:27,331 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 4f4fae27eb4240fbb41650c64006ec08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680036924423 2023-03-28 20:55:27,332 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 78060e479d83459b853d3646b0d4b1b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680036925585 2023-03-28 20:55:27,338 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#C#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:27,366 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/45a9104d3a6e450a86d11f26f6f8d51f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/45a9104d3a6e450a86d11f26f6f8d51f 2023-03-28 20:55:27,369 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/A of 1d7220861206a2835f715e6061645bc2 into 45a9104d3a6e450a86d11f26f6f8d51f(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:27,369 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,369 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/A, priority=13, startTime=1680036926826; duration=0sec 2023-03-28 20:55:27,369 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:27,410 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,413 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,416 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@614da6a0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,423 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,424 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,426 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,427 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,429 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,430 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,434 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,434 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,535 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,538 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,540 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:27,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,544 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7e8793[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,549 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,552 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,552 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,558 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,558 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,658 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,662 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,663 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,663 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,664 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,666 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:27,668 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fee3638[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:27,670 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:27,670 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:27,672 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,672 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:27,675 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:27,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:27,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:27,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:27,676 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:27,677 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,677 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,677 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,678 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:27,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,682 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:27,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:27,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,682 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:27,682 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:27,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:27,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:27,747 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/08310dfe30dc437cb87b1ba808542a3d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/08310dfe30dc437cb87b1ba808542a3d 2023-03-28 20:55:27,750 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1d7220861206a2835f715e6061645bc2/C of 1d7220861206a2835f715e6061645bc2 into 08310dfe30dc437cb87b1ba808542a3d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:27,750 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:27,751 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/C, priority=12, startTime=1680036926826; duration=0sec 2023-03-28 20:55:27,751 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:27,783 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:27,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:27,792 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:27,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:27,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:27,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,797 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:27,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:27,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:27,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:27,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:27,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:27,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:27,800 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:27,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036927842, max timestamp=1680036927849Num uniques 859; 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036927842, max timestamp=1680036927849Num uniques 859; 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036927842, max timestamp=1680036927849Num uniques 859; 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036927862, max timestamp=1680036927878Num uniques 1018; 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036927862, max timestamp=1680036927878Num uniques 1018; 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036927862, max timestamp=1680036927878Num uniques 1018; 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:27,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-28 20:55:27,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036927900, max timestamp=1680036927923Num uniques 1027; 2023-03-28 20:55:27,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036927900, max timestamp=1680036927923Num uniques 1027; 2023-03-28 20:55:27,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-28 20:55:27,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036927900, max timestamp=1680036927923Num uniques 1027; 2023-03-28 20:55:27,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-28 20:55:27,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680036927946, max timestamp=1680036927958Num uniques 1086; 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680036927946, max timestamp=1680036927958Num uniques 1086; 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680036927946, max timestamp=1680036927958Num uniques 1086; 2023-03-28 20:55:27,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680036927982, max timestamp=1680036927995Num uniques 1145; 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680036927982, max timestamp=1680036927995Num uniques 1145; 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680036927982, max timestamp=1680036927995Num uniques 1145; 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-28 20:55:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-28 20:55:28,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:28,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036988026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680036928017, max timestamp=1680036928028Num uniques 1154; 2023-03-28 20:55:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-28 20:55:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680036928017, max timestamp=1680036928028Num uniques 1154; 2023-03-28 20:55:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-28 20:55:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680036928017, max timestamp=1680036928028Num uniques 1154; 2023-03-28 20:55:28,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-28 20:55:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036928064, max timestamp=1680036928075Num uniques 1063; 2023-03-28 20:55:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-28 20:55:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036928064, max timestamp=1680036928075Num uniques 1063; 2023-03-28 20:55:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-28 20:55:28,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036928064, max timestamp=1680036928075Num uniques 1063; 2023-03-28 20:55:28,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-28 20:55:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036928082, max timestamp=1680036928137Num uniques 1072; 2023-03-28 20:55:28,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:55:28,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036928082, max timestamp=1680036928137Num uniques 1072; 2023-03-28 20:55:28,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:55:28,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036928082, max timestamp=1680036928137Num uniques 1072; 2023-03-28 20:55:28,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:55:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036928179, max timestamp=1680036928188Num uniques 1081; 2023-03-28 20:55:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:55:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036928179, max timestamp=1680036928188Num uniques 1081; 2023-03-28 20:55:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036928179, max timestamp=1680036928188Num uniques 1081; 2023-03-28 20:55:28,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:55:28,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:55:28,214 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303281acfd1d5613d4ee08ba6c2668a9d5896 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303281acfd1d5613d4ee08ba6c2668a9d5896 2023-03-28 20:55:28,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680036928219, max timestamp=1680036928248Num uniques 1040; 2023-03-28 20:55:28,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-28 20:55:28,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680036928219, max timestamp=1680036928248Num uniques 1040; 2023-03-28 20:55:28,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-28 20:55:28,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680036928219, max timestamp=1680036928248Num uniques 1040; 2023-03-28 20:55:28,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680036928272, max timestamp=1680036928287Num uniques 1049; 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680036928272, max timestamp=1680036928287Num uniques 1049; 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680036928280, max timestamp=1680036928287Num uniques 1049; 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-28 20:55:28,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680036928346, max timestamp=1680036928365Num uniques 1108; 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680036928346, max timestamp=1680036928365Num uniques 1108; 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680036928346, max timestamp=1680036928365Num uniques 1058; 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-03-28 20:55:28,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=14100, suffix heap size=138000,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=308 2023-03-28 20:55:28,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680036928394, max timestamp=1680036928406Num uniques 1167; 2023-03-28 20:55:28,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-03-28 20:55:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680036928394, max timestamp=1680036928406Num uniques 1167; 2023-03-28 20:55:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-03-28 20:55:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680036928394, max timestamp=1680036928406Num uniques 1167; 2023-03-28 20:55:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036928418, max timestamp=1680036928441Num uniques 1076; 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036928418, max timestamp=1680036928441Num uniques 1076; 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036928418, max timestamp=1680036928441Num uniques 1076; 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-28 20:55:28,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680036928455, max timestamp=1680036928471Num uniques 1035; 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680036928455, max timestamp=1680036928471Num uniques 1035; 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680036928455, max timestamp=1680036928471Num uniques 1035; 2023-03-28 20:55:28,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-28 20:55:28,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680036928488, max timestamp=1680036928500Num uniques 1044; 2023-03-28 20:55:28,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-28 20:55:28,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680036928488, max timestamp=1680036928500Num uniques 1044; 2023-03-28 20:55:28,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-28 20:55:28,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680036928488, max timestamp=1680036928500Num uniques 1044; 2023-03-28 20:55:28,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-28 20:55:28,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680036928526, max timestamp=1680036928541Num uniques 1103; 2023-03-28 20:55:28,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-03-28 20:55:28,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680036928526, max timestamp=1680036928541Num uniques 1103; 2023-03-28 20:55:28,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680036928526, max timestamp=1680036928541Num uniques 1103; 2023-03-28 20:55:28,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-03-28 20:55:28,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-03-28 20:55:28,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1680036928564, max timestamp=1680036928580Num uniques 1112; 2023-03-28 20:55:28,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1680036928564, max timestamp=1680036928580Num uniques 1112; 2023-03-28 20:55:28,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-03-28 20:55:28,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-03-28 20:55:28,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1680036928564, max timestamp=1680036928580Num uniques 1112; 2023-03-28 20:55:28,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-03-28 20:55:28,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680036928601, max timestamp=1680036928613Num uniques 1171; 2023-03-28 20:55:28,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-03-28 20:55:28,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680036928601, max timestamp=1680036928613Num uniques 1171; 2023-03-28 20:55:28,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680036928601, max timestamp=1680036928613Num uniques 1171; 2023-03-28 20:55:28,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-03-28 20:55:28,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-03-28 20:55:28,636 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=402, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/0ff92f3d3600455880326969ed3f22bf 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1680036928640, max timestamp=1680036928654Num uniques 1030; 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1680036928640, max timestamp=1680036928654Num uniques 1030; 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=15110, suffix heap size=135088,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=330 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=15110, suffix heap size=135088,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=330 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1680036928640, max timestamp=1680036928654Num uniques 1030; 2023-03-28 20:55:28,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=15110, suffix heap size=135088,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=330 2023-03-28 20:55:28,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680036928681, max timestamp=1680036928705Num uniques 1189; 2023-03-28 20:55:28,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=10944, suffix heap size=151624,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=239 2023-03-28 20:55:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680036928681, max timestamp=1680036928705Num uniques 1189; 2023-03-28 20:55:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680036928681, max timestamp=1680036928705Num uniques 1189; 2023-03-28 20:55:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=10944, suffix heap size=151624,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=239 2023-03-28 20:55:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=10944, suffix heap size=151624,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=239 2023-03-28 20:55:28,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680036928727, max timestamp=1680036928747Num uniques 1098; 2023-03-28 20:55:28,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-28 20:55:28,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680036928727, max timestamp=1680036928747Num uniques 1098; 2023-03-28 20:55:28,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680036928727, max timestamp=1680036928747Num uniques 1098; 2023-03-28 20:55:28,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-28 20:55:28,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-28 20:55:28,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1680036928769, max timestamp=1680036928784Num uniques 1057; 2023-03-28 20:55:28,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1680036928769, max timestamp=1680036928784Num uniques 1057; 2023-03-28 20:55:28,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-28 20:55:28,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-28 20:55:28,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1680036928769, max timestamp=1680036928784Num uniques 1057; 2023-03-28 20:55:28,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-28 20:55:28,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680036928814, max timestamp=1680036928834Num uniques 1066; 2023-03-28 20:55:28,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-28 20:55:28,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680036928814, max timestamp=1680036928834Num uniques 1066; 2023-03-28 20:55:28,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-28 20:55:28,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680036928814, max timestamp=1680036928834Num uniques 1066; 2023-03-28 20:55:28,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-28 20:55:28,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1680036928855, max timestamp=1680036928870Num uniques 1075; 2023-03-28 20:55:28,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-03-28 20:55:28,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1680036928855, max timestamp=1680036928870Num uniques 1075; 2023-03-28 20:55:28,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-03-28 20:55:28,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1680036928855, max timestamp=1680036928870Num uniques 1075; 2023-03-28 20:55:28,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680036928886, max timestamp=1680036928907Num uniques 1034; 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680036928886, max timestamp=1680036928907Num uniques 1034; 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680036928886, max timestamp=1680036928907Num uniques 1034; 2023-03-28 20:55:28,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-03-28 20:55:28,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1680036928930, max timestamp=1680036928945Num uniques 1093; 2023-03-28 20:55:28,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-28 20:55:28,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1680036928930, max timestamp=1680036928945Num uniques 1093; 2023-03-28 20:55:28,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-28 20:55:28,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1680036928930, max timestamp=1680036928945Num uniques 1093; 2023-03-28 20:55:28,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-28 20:55:28,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680036928977, max timestamp=1680036928992Num uniques 1152; 2023-03-28 20:55:28,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=11540, suffix heap size=147776,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=252 2023-03-28 20:55:28,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680036928977, max timestamp=1680036928992Num uniques 1152; 2023-03-28 20:55:28,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=11540, suffix heap size=147776,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=252 2023-03-28 20:55:28,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1680036928977, max timestamp=1680036928992Num uniques 1152; 2023-03-28 20:55:28,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=11540, suffix heap size=147776,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=252 2023-03-28 20:55:29,045 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/77d308e64e0a4df69206f62a106397f3 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680036929029, max timestamp=1680036929046Num uniques 1111; 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680036929029, max timestamp=1680036929046Num uniques 1111; 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=9656, suffix heap size=143512,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=211 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680036929029, max timestamp=1680036929046Num uniques 1111; 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=9656, suffix heap size=143512,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=211 2023-03-28 20:55:29,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=9656, suffix heap size=143512,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=211 2023-03-28 20:55:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7770, totalHeapSize=17920, min timestamp=1680036929072, max timestamp=1680036929081Num uniques 1070; 2023-03-28 20:55:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48990, new segment data size=7770, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-03-28 20:55:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7777, totalHeapSize=17920, min timestamp=1680036929072, max timestamp=1680036929081Num uniques 1070; 2023-03-28 20:55:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48997, new segment data size=7777, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-03-28 20:55:29,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036929072, max timestamp=1680036929081Num uniques 1070; 2023-03-28 20:55:29,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48992, new segment data size=6870, suffix heap size=139248,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=150 2023-03-28 20:55:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680036929107, max timestamp=1680036929126Num uniques 1029; 2023-03-28 20:55:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-03-28 20:55:29,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680036929107, max timestamp=1680036929126Num uniques 1029; 2023-03-28 20:55:29,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-03-28 20:55:29,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680036929107, max timestamp=1680036929126Num uniques 1009; 2023-03-28 20:55:29,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46222, new segment data size=10484, suffix heap size=132904,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=229 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680036929146, max timestamp=1680036929161Num uniques 1088; 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680036929146, max timestamp=1680036929161Num uniques 1088; 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680036929146, max timestamp=1680036929161Num uniques 1088; 2023-03-28 20:55:29,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680036929172, max timestamp=1680036929190Num uniques 1097; 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680036929172, max timestamp=1680036929190Num uniques 1097; 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=13602, suffix heap size=142056,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=297 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1680036929172, max timestamp=1680036929190Num uniques 1097; 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=13602, suffix heap size=142056,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=297 2023-03-28 20:55:29,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=13602, suffix heap size=142056,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=297 2023-03-28 20:55:29,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036989226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1680036929216, max timestamp=1680036929234Num uniques 1156; 2023-03-28 20:55:29,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=9430, suffix heap size=148192,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=206 2023-03-28 20:55:29,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1680036929216, max timestamp=1680036929234Num uniques 1156; 2023-03-28 20:55:29,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1680036929216, max timestamp=1680036929234Num uniques 1156; 2023-03-28 20:55:29,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=9430, suffix heap size=148192,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=206 2023-03-28 20:55:29,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=9430, suffix heap size=148192,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=206 2023-03-28 20:55:29,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680036929265, max timestamp=1680036929273Num uniques 1065; 2023-03-28 20:55:29,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680036929265, max timestamp=1680036929273Num uniques 1065; 2023-03-28 20:55:29,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1680036929265, max timestamp=1680036929273Num uniques 1065; 2023-03-28 20:55:29,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=9840, suffix heap size=138728,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=215 2023-03-28 20:55:29,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=9840, suffix heap size=138728,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=215 2023-03-28 20:55:29,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=9840, suffix heap size=138728,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=215 2023-03-28 20:55:29,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680036929297, max timestamp=1680036929329Num uniques 1074; 2023-03-28 20:55:29,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-03-28 20:55:29,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680036929297, max timestamp=1680036929329Num uniques 1074; 2023-03-28 20:55:29,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1680036929297, max timestamp=1680036929329Num uniques 1074; 2023-03-28 20:55:29,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-03-28 20:55:29,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1680036929354, max timestamp=1680036929369Num uniques 1083; 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1680036929354, max timestamp=1680036929369Num uniques 1083; 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1680036929354, max timestamp=1680036929369Num uniques 1083; 2023-03-28 20:55:29,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1680036929376, max timestamp=1680036929398Num uniques 1042; 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1680036929376, max timestamp=1680036929398Num uniques 1042; 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1680036929376, max timestamp=1680036929398Num uniques 1042; 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-03-28 20:55:29,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-03-28 20:55:29,454 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/b95c7dc504fe4c9494e9ae4873f85522 2023-03-28 20:55:29,458 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/0ff92f3d3600455880326969ed3f22bf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf 2023-03-28 20:55:29,461 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf, entries=150, sequenceid=402, filesize=25.4 K 2023-03-28 20:55:29,462 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/77d308e64e0a4df69206f62a106397f3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3 2023-03-28 20:55:29,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036989459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1680036929451, max timestamp=1680036929462Num uniques 1151; 2023-03-28 20:55:29,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=6915, suffix heap size=147672,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=151 2023-03-28 20:55:29,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1680036929451, max timestamp=1680036929462Num uniques 1151; 2023-03-28 20:55:29,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=6915, suffix heap size=147672,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=151 2023-03-28 20:55:29,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1680036929451, max timestamp=1680036929462Num uniques 1151; 2023-03-28 20:55:29,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=6915, suffix heap size=147672,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=151 2023-03-28 20:55:29,468 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3, entries=100, sequenceid=402, filesize=9.5 K 2023-03-28 20:55:29,468 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/b95c7dc504fe4c9494e9ae4873f85522 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522 2023-03-28 20:55:29,471 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522, entries=100, sequenceid=402, filesize=9.5 K 2023-03-28 20:55:29,472 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=46.96 KB/48090 for 1d7220861206a2835f715e6061645bc2 in 1672ms, sequenceid=402, compaction requested=true 2023-03-28 20:55:29,472 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:29,472 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:29,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=53.67 KB heapSize=146.67 KB 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730ad49[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:29,479 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:29,479 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,481 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:29,482 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:29,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:29,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:29,485 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:29,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:29,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:29,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:29,486 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:29,487 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,487 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,487 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,489 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:29,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:29,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:29,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,491 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:29,491 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:29,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036929511, max timestamp=1680036929533Num uniques 859; 2023-03-28 20:55:29,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036929511, max timestamp=1680036929533Num uniques 859; 2023-03-28 20:55:29,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:29,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036929511, max timestamp=1680036929533Num uniques 859; 2023-03-28 20:55:29,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:29,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:29,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036989576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036989578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036989584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,591 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:29,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:29,594 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:29,594 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,594 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:29,594 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:29,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,597 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,597 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:29,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:29,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:29,599 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,599 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:29,599 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:29,599 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:29,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:29,602 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc473a9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:29,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:29,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,605 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:29,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:29,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:29,607 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:29,609 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:29,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:29,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:29,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:29,610 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:29,612 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,612 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,612 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:29,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,617 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:29,617 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:29,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036989685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036989685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036989689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,718 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:29,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:29,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:29,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:29,721 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:29,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:29,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:29,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,722 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:29,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:29,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:29,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,726 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,726 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:29,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:29,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212c3f5d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:29,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:29,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:29,733 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:29,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:29,735 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:29,738 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:29,738 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:29,738 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:29,738 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:29,738 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:29,740 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,740 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,740 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,741 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:29,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,747 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,747 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:29,747 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:29,847 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:29,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:29,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:29,850 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:29,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:29,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:29,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,852 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,852 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:29,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:29,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:29,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:29,854 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:29,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:29,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:29,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fed92b6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:29,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:29,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,861 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:29,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,862 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,864 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:29,866 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:29,866 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:29,866 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:29,866 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:29,866 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:29,868 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,868 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,868 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:29,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,868 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,869 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,871 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:29,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,872 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:29,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:29,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:29,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:29,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:29,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:29,872 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:29,872 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:29,886 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032893357b927bb445348b93bc7deae0435a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893357b927bb445348b93bc7deae0435a 2023-03-28 20:55:29,891 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1080, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/36231bf84028435296c975d1eacab11a 2023-03-28 20:55:29,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036989893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036989893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:29,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036989893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1080 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/40d0fcf38d144cdbbf281cd2f4fe4620 2023-03-28 20:55:29,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1080 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/d0078e4054364777834fcb1634d18d6a 2023-03-28 20:55:29,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/36231bf84028435296c975d1eacab11a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a 2023-03-28 20:55:29,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a, entries=150, sequenceid=1080, filesize=25.4 K 2023-03-28 20:55:29,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/40d0fcf38d144cdbbf281cd2f4fe4620 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620 2023-03-28 20:55:29,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620, entries=150, sequenceid=1080, filesize=11.9 K 2023-03-28 20:55:29,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/d0078e4054364777834fcb1634d18d6a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a 2023-03-28 20:55:29,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a, entries=150, sequenceid=1080, filesize=11.9 K 2023-03-28 20:55:29,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~146.63 KB/150144, currentSize=134.18 KB/137400 for 1d7220861206a2835f715e6061645bc2 in 466ms, sequenceid=1080, compaction requested=true 2023-03-28 20:55:29,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:29,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:29,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:29,940 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:29,940 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:29,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:29,941 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104760 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:29,941 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/A is initiating minor compaction (all files) 2023-03-28 20:55:29,941 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:29,941 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/A in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,941 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/B is initiating minor compaction (all files) 2023-03-28 20:55:29,941 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/45a9104d3a6e450a86d11f26f6f8d51f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=102.3 K 2023-03-28 20:55:29,941 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/B in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,941 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6dd36f85a31743e3a01fa4459497054d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=45.8 K 2023-03-28 20:55:29,942 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 45a9104d3a6e450a86d11f26f6f8d51f, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680036924423 2023-03-28 20:55:29,942 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6dd36f85a31743e3a01fa4459497054d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680036924423 2023-03-28 20:55:29,942 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b47ee558e789490c88641d926792dbc5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680036925585 2023-03-28 20:55:29,942 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1838b65316654f29976fb5ea473729f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680036925585 2023-03-28 20:55:29,942 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0ff92f3d3600455880326969ed3f22bf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680036927768 2023-03-28 20:55:29,943 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 77d308e64e0a4df69206f62a106397f3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680036927768 2023-03-28 20:55:29,943 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 36231bf84028435296c975d1eacab11a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1680036929459 2023-03-28 20:55:29,943 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 40d0fcf38d144cdbbf281cd2f4fe4620, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1680036929459 2023-03-28 20:55:29,972 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#B#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:29,973 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:29,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:29,977 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#A#compaction#431 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:29,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:29,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:29,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:29,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:29,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:29,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:29,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:29,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:29,996 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:29,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:30,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:30,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:30,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036990213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036990214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036990216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036990317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036990321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036990321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,391 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/1b1cde81fcb44e5cb40f4210150e1395 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1b1cde81fcb44e5cb40f4210150e1395 2023-03-28 20:55:30,396 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1d7220861206a2835f715e6061645bc2/B of 1d7220861206a2835f715e6061645bc2 into 1b1cde81fcb44e5cb40f4210150e1395(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:30,396 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:30,396 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/B, priority=12, startTime=1680036929940; duration=0sec 2023-03-28 20:55:30,397 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:30,397 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:30,398 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34728 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:30,398 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1d7220861206a2835f715e6061645bc2/C is initiating minor compaction (all files) 2023-03-28 20:55:30,398 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1d7220861206a2835f715e6061645bc2/C in TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:30,398 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/08310dfe30dc437cb87b1ba808542a3d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp, totalSize=33.9 K 2023-03-28 20:55:30,398 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 08310dfe30dc437cb87b1ba808542a3d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680036925585 2023-03-28 20:55:30,398 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b95c7dc504fe4c9494e9ae4873f85522, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680036927768 2023-03-28 20:55:30,399 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d0078e4054364777834fcb1634d18d6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1680036929459 2023-03-28 20:55:30,411 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1d7220861206a2835f715e6061645bc2#C#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:30,413 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032870e43b3fa5a34f938677a8fac9240641 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032870e43b3fa5a34f938677a8fac9240641 2023-03-28 20:55:30,428 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/7616e1306b1747e6b6c618bd6938d4e8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7616e1306b1747e6b6c618bd6938d4e8 2023-03-28 20:55:30,428 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1116, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/27f095c31522495dbd5a14c3f2243990 2023-03-28 20:55:30,432 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1d7220861206a2835f715e6061645bc2/C of 1d7220861206a2835f715e6061645bc2 into 7616e1306b1747e6b6c618bd6938d4e8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:30,432 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:30,432 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/C, priority=13, startTime=1680036929940; duration=0sec 2023-03-28 20:55:30,433 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:30,437 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1116 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/d61810f29526403a8880023cb64eafa3 2023-03-28 20:55:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036990522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036990524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036990525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036990828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036990828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:30,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036990829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,846 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1116 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a6227a8cfc7c48f9b98cc4dd09480465 2023-03-28 20:55:30,849 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/27f095c31522495dbd5a14c3f2243990 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/27f095c31522495dbd5a14c3f2243990 2023-03-28 20:55:30,852 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/27f095c31522495dbd5a14c3f2243990, entries=150, sequenceid=1116, filesize=25.4 K 2023-03-28 20:55:30,853 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/d61810f29526403a8880023cb64eafa3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d61810f29526403a8880023cb64eafa3 2023-03-28 20:55:30,858 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d61810f29526403a8880023cb64eafa3, entries=150, sequenceid=1116, filesize=11.9 K 2023-03-28 20:55:30,859 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a6227a8cfc7c48f9b98cc4dd09480465 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a6227a8cfc7c48f9b98cc4dd09480465 2023-03-28 20:55:30,865 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a6227a8cfc7c48f9b98cc4dd09480465, entries=150, sequenceid=1116, filesize=11.9 K 2023-03-28 20:55:30,866 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1d7220861206a2835f715e6061645bc2 in 870ms, sequenceid=1116, compaction requested=false 2023-03-28 20:55:30,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:30,866 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:30,869 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,869 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d353ae7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:30,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:30,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,873 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:30,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:30,875 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:30,875 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:30,877 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,877 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,877 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:30,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:30,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:30,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,883 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,884 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,884 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:30,884 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:30,884 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:30,984 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:30,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:30,988 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:30,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:30,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:30,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:30,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:30,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:30,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:30,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:30,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:30,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:30,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:30,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. started... 2023-03-28 20:55:30,991 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:30,992 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:31,015 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032855b32e05b862441ebcf75f493236180c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032855b32e05b862441ebcf75f493236180c 2023-03-28 20:55:31,041 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1128, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e26c05a2cde64f01bc3ab81d31f05116 2023-03-28 20:55:31,227 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/5b7bfdcca1554520aeb8b794c0abe5ac as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5b7bfdcca1554520aeb8b794c0abe5ac 2023-03-28 20:55:31,230 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1d7220861206a2835f715e6061645bc2/A of 1d7220861206a2835f715e6061645bc2 into 5b7bfdcca1554520aeb8b794c0abe5ac(size=26.1 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-28 20:55:31,230 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:31,230 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2., storeName=1d7220861206a2835f715e6061645bc2/A, priority=12, startTime=1680036929939; duration=1sec 2023-03-28 20:55:31,230 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:31,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:31,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. as already flushing 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036931350, max timestamp=1680036931369Num uniques 859; 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036931350, max timestamp=1680036931369Num uniques 859; 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036931350, max timestamp=1680036931369Num uniques 859; 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:31,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:31,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036991426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036991426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 517 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036991433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,479 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/c92fba8a067948b08f5a547e7179e6c6 2023-03-28 20:55:31,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036991535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036991538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 519 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036991541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48210 deadline: 1680036991740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48194 deadline: 1680036991743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:31,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 521 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680036991746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,850 DEBUG [Thread-2693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27313b38 to 127.0.0.1:56827 2023-03-28 20:55:31,850 DEBUG [Thread-2693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:31,855 DEBUG [Thread-2687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x313c7ad6 to 127.0.0.1:56827 2023-03-28 20:55:31,855 DEBUG [Thread-2687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:31,857 DEBUG [Thread-2691] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x750d2a7d to 127.0.0.1:56827 2023-03-28 20:55:31,857 DEBUG [Thread-2691] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:31,857 DEBUG [Thread-2689] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20d28b68 to 127.0.0.1:56827 2023-03-28 20:55:31,857 DEBUG [Thread-2689] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:31,863 DEBUG [Thread-2685] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d4bfeca to 127.0.0.1:56827 2023-03-28 20:55:31,863 DEBUG [Thread-2685] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:31,888 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c52ecfc16f55406896c86864e75c9330 2023-03-28 20:55:31,891 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e26c05a2cde64f01bc3ab81d31f05116 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e26c05a2cde64f01bc3ab81d31f05116 2023-03-28 20:55:31,894 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e26c05a2cde64f01bc3ab81d31f05116, entries=150, sequenceid=1128, filesize=25.4 K 2023-03-28 20:55:31,894 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/c92fba8a067948b08f5a547e7179e6c6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c92fba8a067948b08f5a547e7179e6c6 2023-03-28 20:55:31,897 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c92fba8a067948b08f5a547e7179e6c6, entries=150, sequenceid=1128, filesize=11.9 K 2023-03-28 20:55:31,898 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/c52ecfc16f55406896c86864e75c9330 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c52ecfc16f55406896c86864e75c9330 2023-03-28 20:55:31,900 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c52ecfc16f55406896c86864e75c9330, entries=150, sequenceid=1128, filesize=11.9 K 2023-03-28 20:55:31,901 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1d7220861206a2835f715e6061645bc2 in 910ms, sequenceid=1128, compaction requested=true 2023-03-28 20:55:31,901 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:31,901 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:31,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:31,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:31,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:31,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c45fd3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:31,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:31,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,908 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:31,908 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,908 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:31,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:31,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:31,911 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:31,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:31,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:31,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:31,911 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:31,913 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:31,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:31,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:31,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,914 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:31,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:31,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:31,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:31,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:31,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:31,917 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:31,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,917 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:31,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:31,917 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:31,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:32,017 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:32,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:32,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:32,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-28 20:55:32,045 DEBUG [Thread-2680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x455a88b1 to 127.0.0.1:56827 2023-03-28 20:55:32,045 DEBUG [Thread-2680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:32,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:32,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:32,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:32,049 DEBUG [Thread-2674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cee66b5 to 127.0.0.1:56827 2023-03-28 20:55:32,050 DEBUG [Thread-2674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,050 DEBUG [Thread-2676] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c1d19b3 to 127.0.0.1:56827 2023-03-28 20:55:32,050 DEBUG [Thread-2676] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,439 DEBUG [Thread-2682] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x722b2407 to 127.0.0.1:56827 2023-03-28 20:55:32,439 DEBUG [Thread-2682] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,457 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303289d4b278559814b1787435f34b0348bfa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289d4b278559814b1787435f34b0348bfa 2023-03-28 20:55:32,526 DEBUG [Thread-2678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14e151fb to 127.0.0.1:56827 2023-03-28 20:55:32,527 DEBUG [Thread-2678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 414 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 310 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 306 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1777 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5329 rows 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1781 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5343 rows 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1788 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5364 rows 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1779 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5337 rows 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 1795 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 5385 rows 2023-03-28 20:55:32,527 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:55:32,528 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67b6d133 to 127.0.0.1:56827 2023-03-28 20:55:32,528 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:32,529 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:55:32,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:55:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:32,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:32,533 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036932533"}]},"ts":"1680036932533"} 2023-03-28 20:55:32,535 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:55:32,536 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:55:32,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, UNASSIGN}] 2023-03-28 20:55:32,538 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, UNASSIGN 2023-03-28 20:55:32,538 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:32,538 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036932538"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036932538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036932538"}]},"ts":"1680036932538"} 2023-03-28 20:55:32,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:32,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:32,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1d7220861206a2835f715e6061645bc2, disabling compactions & flushes 2023-03-28 20:55:32,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:32,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:32,861 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1167, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e852cf26ff554d3db1d6ba55b22620c3 2023-03-28 20:55:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:33,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1167 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/8eb808c6b481498fa01404f0cfbdd9d4 2023-03-28 20:55:33,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:33,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1167 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a44d8be9009e4152b837c3b85fd95bec 2023-03-28 20:55:33,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/e852cf26ff554d3db1d6ba55b22620c3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e852cf26ff554d3db1d6ba55b22620c3 2023-03-28 20:55:33,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e852cf26ff554d3db1d6ba55b22620c3, entries=150, sequenceid=1167, filesize=25.4 K 2023-03-28 20:55:33,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/8eb808c6b481498fa01404f0cfbdd9d4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8eb808c6b481498fa01404f0cfbdd9d4 2023-03-28 20:55:33,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8eb808c6b481498fa01404f0cfbdd9d4, entries=150, sequenceid=1167, filesize=11.9 K 2023-03-28 20:55:33,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/a44d8be9009e4152b837c3b85fd95bec as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a44d8be9009e4152b837c3b85fd95bec 2023-03-28 20:55:33,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a44d8be9009e4152b837c3b85fd95bec, entries=150, sequenceid=1167, filesize=11.9 K 2023-03-28 20:55:33,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=26.84 KB/27480 for 1d7220861206a2835f715e6061645bc2 in 1645ms, sequenceid=1167, compaction requested=true 2023-03-28 20:55:33,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:33,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:33,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:33,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:33,690 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. because compaction request was cancelled 2023-03-28 20:55:33,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:33,690 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. because compaction request was cancelled 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. after waiting 0 ms 2023-03-28 20:55:33,690 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. because compaction request was cancelled 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:33,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1d7220861206a2835f715e6061645bc2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=A 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=B 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1d7220861206a2835f715e6061645bc2, store=C 2023-03-28 20:55:33,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:34,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328380a4937594c4d999d423d93f47b38b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328380a4937594c4d999d423d93f47b38b7 2023-03-28 20:55:34,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1174, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/b5b09b23978843db94cf384956788911 2023-03-28 20:55:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:34,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/15db5b75d1ab44f781463e5defebfea6 2023-03-28 20:55:35,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/00f4b9de7b794f9c9fda7fd4ce01b540 2023-03-28 20:55:35,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/A/b5b09b23978843db94cf384956788911 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b5b09b23978843db94cf384956788911 2023-03-28 20:55:35,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b5b09b23978843db94cf384956788911, entries=100, sequenceid=1174, filesize=18.6 K 2023-03-28 20:55:35,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/B/15db5b75d1ab44f781463e5defebfea6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/15db5b75d1ab44f781463e5defebfea6 2023-03-28 20:55:35,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/15db5b75d1ab44f781463e5defebfea6, entries=100, sequenceid=1174, filesize=9.5 K 2023-03-28 20:55:35,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/.tmp/C/00f4b9de7b794f9c9fda7fd4ce01b540 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/00f4b9de7b794f9c9fda7fd4ce01b540 2023-03-28 20:55:35,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/00f4b9de7b794f9c9fda7fd4ce01b540, entries=100, sequenceid=1174, filesize=9.5 K 2023-03-28 20:55:35,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1d7220861206a2835f715e6061645bc2 in 1646ms, sequenceid=1174, compaction requested=true 2023-03-28 20:55:35,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c56e8ac12ddf42cda5e49fe8b83e08f4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/aa032192b498430fa4e9cda1dd8e2fc6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ae3ed6a30da84613aee8feb23b94b09b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/45a9104d3a6e450a86d11f26f6f8d51f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a] to archive 2023-03-28 20:55:35,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:35,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ff500f6f854040c4b6159f15656ec5fd 2023-03-28 20:55:35,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ea98ca487a4a48f1b2fa40c45747deb8 2023-03-28 20:55:35,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4f4a7d19471a4b368401bca287531e53 2023-03-28 20:55:35,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e4ecc313a9f94fed9f4e9afe7c9acb64 2023-03-28 20:55:35,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c91fa6a8d1e541ce93769a72c8d26348 2023-03-28 20:55:35,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cd074809b3924c7781d55ee1f4b9a026 2023-03-28 20:55:35,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c56e8ac12ddf42cda5e49fe8b83e08f4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/c56e8ac12ddf42cda5e49fe8b83e08f4 2023-03-28 20:55:35,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/a01578119fb04edd84ed18aae6c46381 2023-03-28 20:55:35,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/48fe294173554e57abb8f5ef2a435043 2023-03-28 20:55:35,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/330fb402024c48ffb83c34b8ad832a2a 2023-03-28 20:55:35,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/4add1a10232745bc81fd68b201baafe9 2023-03-28 20:55:35,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/aa032192b498430fa4e9cda1dd8e2fc6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/aa032192b498430fa4e9cda1dd8e2fc6 2023-03-28 20:55:35,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5de2f0f4508f4f03bd0d922f81e05818 2023-03-28 20:55:35,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e45f47d492494264a477a9d54f17d3eb 2023-03-28 20:55:35,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ae3ed6a30da84613aee8feb23b94b09b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/ae3ed6a30da84613aee8feb23b94b09b 2023-03-28 20:55:35,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/cb4544b66a084f9198353d8357926bfc 2023-03-28 20:55:35,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/de37a3a6dc504db7ba558c1af6af79d0 2023-03-28 20:55:35,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/45a9104d3a6e450a86d11f26f6f8d51f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/45a9104d3a6e450a86d11f26f6f8d51f 2023-03-28 20:55:35,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b47ee558e789490c88641d926792dbc5 2023-03-28 20:55:35,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/0ff92f3d3600455880326969ed3f22bf 2023-03-28 20:55:35,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/36231bf84028435296c975d1eacab11a 2023-03-28 20:55:35,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/25776ff8967444fdb00df4d7ed330910, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/0fa405ba0a1d4a888a33e0de8932bc30, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6410406308224466b171891eb65c35f9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6dd36f85a31743e3a01fa4459497054d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620] to archive 2023-03-28 20:55:35,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:35,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/497afdf67a8e47eb8a8325fc6b2749b8 2023-03-28 20:55:35,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/4ed6912c8bef429ebdf1126f82115553 2023-03-28 20:55:35,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8bc32c7630a44aec8d0c148f0e2db22f 2023-03-28 20:55:35,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d2060068781c47e69ef1a20d50b16cc4 2023-03-28 20:55:35,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/aa39f3eed79840c4900dbaa2651359ed 2023-03-28 20:55:35,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/25776ff8967444fdb00df4d7ed330910 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/25776ff8967444fdb00df4d7ed330910 2023-03-28 20:55:35,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/75413c61d2bb4213aa3af8f0d907b80a 2023-03-28 20:55:35,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/29ed5665db9040679aabaf753c3b79fe 2023-03-28 20:55:35,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1f6c128876bc441c9542810bf2325d8e 2023-03-28 20:55:35,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c03283d5cabe496e92916d1f9fa77994 2023-03-28 20:55:35,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/0fa405ba0a1d4a888a33e0de8932bc30 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/0fa405ba0a1d4a888a33e0de8932bc30 2023-03-28 20:55:35,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/644ef1cfed4e4db2bf71924e896259b1 2023-03-28 20:55:35,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/a0fddeb37cc3406c8a40063e56b0a26e 2023-03-28 20:55:35,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6410406308224466b171891eb65c35f9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6410406308224466b171891eb65c35f9 2023-03-28 20:55:35,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3cfcec97330340ae913a8f26e077c2f7 2023-03-28 20:55:35,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/3839dc12ae244fd1b8132679f6906613 2023-03-28 20:55:35,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6dd36f85a31743e3a01fa4459497054d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/6dd36f85a31743e3a01fa4459497054d 2023-03-28 20:55:35,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/60cd99591d6f48b2b1cfbbc130b845ee 2023-03-28 20:55:35,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1838b65316654f29976fb5ea473729f6 2023-03-28 20:55:35,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/77d308e64e0a4df69206f62a106397f3 2023-03-28 20:55:35,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/40d0fcf38d144cdbbf281cd2f4fe4620 2023-03-28 20:55:35,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/ad1eb766eaef44ccbb831909abcfe870, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b9b78f08366e4d4c818df9952e3bdfb6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/cae0d82a19044a5e8cc1f896b36db774, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/08310dfe30dc437cb87b1ba808542a3d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a] to archive 2023-03-28 20:55:35,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:55:35,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/bda98b77a3e64ea19ad21bd7ae9dab96 2023-03-28 20:55:35,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c39ece258f6d4e2ba19093b8d8052b5f 2023-03-28 20:55:35,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d375f10aa55947e593bdb2d44101a040 2023-03-28 20:55:35,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c216bfb42444f9a867df8dff3644f2f 2023-03-28 20:55:35,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/62968c1358f94096873beeeb6ee0e447 2023-03-28 20:55:35,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/ad1eb766eaef44ccbb831909abcfe870 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/ad1eb766eaef44ccbb831909abcfe870 2023-03-28 20:55:35,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7f6efb91da9148e3ba59033f7368cf48 2023-03-28 20:55:35,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/6c9a28cb7d8f4170a18c42d32afc51fa 2023-03-28 20:55:35,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/322f7354adc04f2eaa0f3925d3b8cb09 2023-03-28 20:55:35,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c2f32914ba5e401d820f87c7871f33ea 2023-03-28 20:55:35,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b9b78f08366e4d4c818df9952e3bdfb6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b9b78f08366e4d4c818df9952e3bdfb6 2023-03-28 20:55:35,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a931b4b2b3494a1b832be6ee3abc81e8 2023-03-28 20:55:35,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/81466c8f96cb4ce1bd1818149983e08a 2023-03-28 20:55:35,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/cae0d82a19044a5e8cc1f896b36db774 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/cae0d82a19044a5e8cc1f896b36db774 2023-03-28 20:55:35,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/75da501de6ea4f69bf46a0687fa40844 2023-03-28 20:55:35,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/9b52d96af3094c23875747e2de586b69 2023-03-28 20:55:35,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/4f4fae27eb4240fbb41650c64006ec08 2023-03-28 20:55:35,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/08310dfe30dc437cb87b1ba808542a3d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/08310dfe30dc437cb87b1ba808542a3d 2023-03-28 20:55:35,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/78060e479d83459b853d3646b0d4b1b7 2023-03-28 20:55:35,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/b95c7dc504fe4c9494e9ae4873f85522 2023-03-28 20:55:35,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/d0078e4054364777834fcb1634d18d6a 2023-03-28 20:55:35,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits/1177.seqid, newMaxSeqId=1177, maxSeqId=4 2023-03-28 20:55:35,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2. 2023-03-28 20:55:35,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1d7220861206a2835f715e6061645bc2: 2023-03-28 20:55:35,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:35,422 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1d7220861206a2835f715e6061645bc2, regionState=CLOSED 2023-03-28 20:55:35,423 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036935422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036935422"}]},"ts":"1680036935422"} 2023-03-28 20:55:35,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-28 20:55:35,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 1d7220861206a2835f715e6061645bc2, server=jenkins-hbase4.apache.org,42693,1680036815956 in 2.8840 sec 2023-03-28 20:55:35,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-28 20:55:35,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1d7220861206a2835f715e6061645bc2, UNASSIGN in 2.8880 sec 2023-03-28 20:55:35,426 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036935426"}]},"ts":"1680036935426"} 2023-03-28 20:55:35,427 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:55:35,431 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:55:35,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9020 sec 2023-03-28 20:55:36,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-28 20:55:36,638 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-28 20:55:36,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:55:36,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,641 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-28 20:55:36,641 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,645 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:36,647 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits] 2023-03-28 20:55:36,649 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/27f095c31522495dbd5a14c3f2243990 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/27f095c31522495dbd5a14c3f2243990 2023-03-28 20:55:36,650 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5b7bfdcca1554520aeb8b794c0abe5ac to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/5b7bfdcca1554520aeb8b794c0abe5ac 2023-03-28 20:55:36,651 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b5b09b23978843db94cf384956788911 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/b5b09b23978843db94cf384956788911 2023-03-28 20:55:36,652 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e26c05a2cde64f01bc3ab81d31f05116 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e26c05a2cde64f01bc3ab81d31f05116 2023-03-28 20:55:36,653 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e852cf26ff554d3db1d6ba55b22620c3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/A/e852cf26ff554d3db1d6ba55b22620c3 2023-03-28 20:55:36,655 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/15db5b75d1ab44f781463e5defebfea6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/15db5b75d1ab44f781463e5defebfea6 2023-03-28 20:55:36,656 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1b1cde81fcb44e5cb40f4210150e1395 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/1b1cde81fcb44e5cb40f4210150e1395 2023-03-28 20:55:36,657 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8eb808c6b481498fa01404f0cfbdd9d4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/8eb808c6b481498fa01404f0cfbdd9d4 2023-03-28 20:55:36,658 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c92fba8a067948b08f5a547e7179e6c6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/c92fba8a067948b08f5a547e7179e6c6 2023-03-28 20:55:36,659 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d61810f29526403a8880023cb64eafa3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/B/d61810f29526403a8880023cb64eafa3 2023-03-28 20:55:36,661 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/00f4b9de7b794f9c9fda7fd4ce01b540 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/00f4b9de7b794f9c9fda7fd4ce01b540 2023-03-28 20:55:36,662 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7616e1306b1747e6b6c618bd6938d4e8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/7616e1306b1747e6b6c618bd6938d4e8 2023-03-28 20:55:36,662 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a44d8be9009e4152b837c3b85fd95bec to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a44d8be9009e4152b837c3b85fd95bec 2023-03-28 20:55:36,663 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a6227a8cfc7c48f9b98cc4dd09480465 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/a6227a8cfc7c48f9b98cc4dd09480465 2023-03-28 20:55:36,664 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c52ecfc16f55406896c86864e75c9330 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/C/c52ecfc16f55406896c86864e75c9330 2023-03-28 20:55:36,666 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits/1177.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2/recovered.edits/1177.seqid 2023-03-28 20:55:36,667 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1d7220861206a2835f715e6061645bc2 2023-03-28 20:55:36,667 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:55:36,667 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:55:36,668 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-28 20:55:36,670 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280c673c2794fc4404960dbccc7be93b52 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303280c673c2794fc4404960dbccc7be93b52 2023-03-28 20:55:36,671 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303281acfd1d5613d4ee08ba6c2668a9d5896 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303281acfd1d5613d4ee08ba6c2668a9d5896 2023-03-28 20:55:36,672 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032825fd7d5ded7d4ba6badafcfa8f425ef3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032825fd7d5ded7d4ba6badafcfa8f425ef3 2023-03-28 20:55:36,673 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328380a4937594c4d999d423d93f47b38b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328380a4937594c4d999d423d93f47b38b7 2023-03-28 20:55:36,674 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284c2adb1f17f64678bf9b2ffd6c25f60f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303284c2adb1f17f64678bf9b2ffd6c25f60f 2023-03-28 20:55:36,674 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032855b32e05b862441ebcf75f493236180c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032855b32e05b862441ebcf75f493236180c 2023-03-28 20:55:36,675 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032863028a08b3d4442e8af828b850141352 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032863028a08b3d4442e8af828b850141352 2023-03-28 20:55:36,676 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286333105e562c4c148f5265d15cafb194 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286333105e562c4c148f5265d15cafb194 2023-03-28 20:55:36,677 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286d62597acf364e9d9b77188789e45d78 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286d62597acf364e9d9b77188789e45d78 2023-03-28 20:55:36,678 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032870e43b3fa5a34f938677a8fac9240641 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032870e43b3fa5a34f938677a8fac9240641 2023-03-28 20:55:36,678 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875a4514367f243c894c63c85b2abc77b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875a4514367f243c894c63c85b2abc77b 2023-03-28 20:55:36,679 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328765b2c1a1efb477d8aafcf35d831c604 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328765b2c1a1efb477d8aafcf35d831c604 2023-03-28 20:55:36,680 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288c30aeb22a4f46c5b0e938df390ff330 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288c30aeb22a4f46c5b0e938df390ff330 2023-03-28 20:55:36,681 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893357b927bb445348b93bc7deae0435a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893357b927bb445348b93bc7deae0435a 2023-03-28 20:55:36,682 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893fdd381ff214615b620a964414afe91 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032893fdd381ff214615b620a964414afe91 2023-03-28 20:55:36,683 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289d4b278559814b1787435f34b0348bfa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289d4b278559814b1787435f34b0348bfa 2023-03-28 20:55:36,684 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb76975b74c3482d91dc6d0e7786d645 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb76975b74c3482d91dc6d0e7786d645 2023-03-28 20:55:36,684 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ccc4516d688549cd8858a0f957bca0c9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ccc4516d688549cd8858a0f957bca0c9 2023-03-28 20:55:36,685 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d0321eee7c81462dae87ca4296bbaff2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d0321eee7c81462dae87ca4296bbaff2 2023-03-28 20:55:36,686 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d3b16dbc9d384a0c98ffbf14d2cf1e32 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328d3b16dbc9d384a0c98ffbf14d2cf1e32 2023-03-28 20:55:36,687 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328e4c82f8c450d4594830e63e75b07bfe4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328e4c82f8c450d4594830e63e75b07bfe4 2023-03-28 20:55:36,687 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:55:36,689 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,690 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:55:36,691 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:55:36,692 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,692 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:55:36,692 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036936692"}]},"ts":"9223372036854775807"} 2023-03-28 20:55:36,693 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:55:36,693 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1d7220861206a2835f715e6061645bc2, NAME => 'TestAcidGuarantees,,1680036909765.1d7220861206a2835f715e6061645bc2.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:55:36,693 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:55:36,693 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036936693"}]},"ts":"9223372036854775807"} 2023-03-28 20:55:36,695 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:55:36,696 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-28 20:55:36,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-28 20:55:36,742 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-28 20:55:36,754 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 225) - Thread LEAK? -, OpenFileDescriptor=563 (was 589), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=365 (was 358) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=6533 (was 6608) 2023-03-28 20:55:36,765 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=171, AvailableMemoryMB=6532 2023-03-28 20:55:36,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:55:36,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:55:36,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:36,769 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:55:36,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-28 20:55:36,769 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:55:36,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-28 20:55:36,770 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:36,770 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 empty. 2023-03-28 20:55:36,771 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:36,771 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:55:36,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-28 20:55:37,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-28 20:55:37,178 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:55:37,179 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 688ef3a4c2e822fe7803c6aa582a3052, NAME => 'TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 688ef3a4c2e822fe7803c6aa582a3052, disabling compactions & flushes 2023-03-28 20:55:37,185 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. after waiting 0 ms 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,185 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,185 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:37,187 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:55:37,188 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036937188"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036937188"}]},"ts":"1680036937188"} 2023-03-28 20:55:37,189 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:55:37,189 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:55:37,189 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036937189"}]},"ts":"1680036937189"} 2023-03-28 20:55:37,190 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:55:37,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, ASSIGN}] 2023-03-28 20:55:37,200 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, ASSIGN 2023-03-28 20:55:37,201 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:55:37,352 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=688ef3a4c2e822fe7803c6aa582a3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:37,352 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036937352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036937352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036937352"}]},"ts":"1680036937352"} 2023-03-28 20:55:37,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:37,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-28 20:55:37,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 688ef3a4c2e822fe7803c6aa582a3052, NAME => 'TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:55:37,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:55:37,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,509 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,510 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A 2023-03-28 20:55:37,510 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A 2023-03-28 20:55:37,510 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:37,510 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 688ef3a4c2e822fe7803c6aa582a3052 columnFamilyName A 2023-03-28 20:55:37,511 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(310): Store=688ef3a4c2e822fe7803c6aa582a3052/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:37,511 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,512 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B 2023-03-28 20:55:37,512 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B 2023-03-28 20:55:37,512 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:37,512 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 688ef3a4c2e822fe7803c6aa582a3052 columnFamilyName B 2023-03-28 20:55:37,513 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(310): Store=688ef3a4c2e822fe7803c6aa582a3052/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:37,513 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,514 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C 2023-03-28 20:55:37,514 DEBUG [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C 2023-03-28 20:55:37,514 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:55:37,514 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 688ef3a4c2e822fe7803c6aa582a3052 columnFamilyName C 2023-03-28 20:55:37,514 INFO [StoreOpener-688ef3a4c2e822fe7803c6aa582a3052-1] regionserver.HStore(310): Store=688ef3a4c2e822fe7803c6aa582a3052/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:55:37,514 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:55:37,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:55:37,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 688ef3a4c2e822fe7803c6aa582a3052; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68027230, jitterRate=0.013684719800949097}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:55:37,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:37,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., pid=49, masterSystemTime=1680036937504 2023-03-28 20:55:37,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,521 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=688ef3a4c2e822fe7803c6aa582a3052, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:37,521 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036937521"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036937521"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036937521"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036937521"}]},"ts":"1680036937521"} 2023-03-28 20:55:37,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-28 20:55:37,523 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 in 168 msec 2023-03-28 20:55:37,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-28 20:55:37,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, ASSIGN in 325 msec 2023-03-28 20:55:37,524 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:55:37,524 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036937524"}]},"ts":"1680036937524"} 2023-03-28 20:55:37,525 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:55:37,527 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:55:37,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 760 msec 2023-03-28 20:55:37,700 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-28 20:55:37,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-28 20:55:37,872 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-28 20:55:37,873 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x18674076 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,878 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c8fbc35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,879 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,881 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58722, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,882 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:55:37,884 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:55:37,885 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1df28b68 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,889 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fb41c50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,889 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5cd71f to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,894 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@357cb237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,895 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x1f0735dc to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,900 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4539ed0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,901 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x14c209bc to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,906 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32cd672d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,906 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x0db388d3 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,909 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b8ff30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,910 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x4c050db8 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,920 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a3a77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,920 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x28e3107b to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,926 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cefb012, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,926 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x14f9adf3 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,930 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24a917df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,930 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x21359a0e to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,935 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78fa3dee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,935 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x3bbac0b5 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:55:37,941 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36d2d826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:55:37,960 DEBUG [hconnection-0x43b0fe87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,960 DEBUG [hconnection-0x5a130fa7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,964 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,965 DEBUG [hconnection-0x68019b50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,965 DEBUG [hconnection-0xce26c93-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,965 DEBUG [hconnection-0x323b2a4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,965 DEBUG [hconnection-0x347c8b06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,968 DEBUG [hconnection-0x1614274a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:37,967 DEBUG [hconnection-0x78ece7d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,967 DEBUG [hconnection-0x4f77690b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,967 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58734, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,967 DEBUG [hconnection-0x30a871e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:55:37,970 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,971 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58802, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,977 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,977 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,977 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58788, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,977 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,980 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58786, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,980 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:55:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:37,988 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:37,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:37,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:37,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:37,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:37,988 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:37,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:37,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:37,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:37,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:37,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:37,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:37,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:37,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:37,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:55:37,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:37,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:37,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:37,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:37,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:37,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:37,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:37,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:37,996 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:37,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:37,996 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,011 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,011 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55524ce1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,014 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,014 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,017 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,017 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036938003, max timestamp=1680036938016Num uniques 859; 2023-03-28 20:55:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036938003, max timestamp=1680036938016Num uniques 859; 2023-03-28 20:55:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:38,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036938003, max timestamp=1680036938016Num uniques 859; 2023-03-28 20:55:38,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:38,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,021 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,021 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,023 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,023 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,027 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,027 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,028 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,028 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680036998035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680036998036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680036998037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680036998038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680036998044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,128 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,138 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,139 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,139 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680036998139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680036998142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680036998146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680036998147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680036998148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,156 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,156 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,160 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,160 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f90cdf[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,164 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,166 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,170 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,170 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,177 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,177 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,177 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,277 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,284 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,285 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128fece5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,307 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,307 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,314 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,314 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680036998345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680036998349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680036998349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680036998349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680036998349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,414 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,417 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/5d87f8808ad1461ba791eb1a6807d187 2023-03-28 20:55:38,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,421 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,423 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@250203a0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,425 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,434 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,436 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,437 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,438 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,438 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,441 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,441 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,442 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,442 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,542 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,545 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,545 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,549 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,550 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,550 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,550 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627076a5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,556 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,558 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,558 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,558 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,566 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,566 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,567 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,569 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,570 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,570 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,570 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,570 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,570 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,570 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680036998651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680036998657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680036998657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680036998657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:38,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680036998657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,670 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,673 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,674 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,674 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,675 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,676 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,677 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,677 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,677 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,677 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,679 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2111c0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,680 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,680 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,687 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,687 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,687 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,691 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,696 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,697 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,697 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,697 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,797 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,800 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,800 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,800 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,800 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,802 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,804 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,804 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,804 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,804 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41242865[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,810 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,810 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,810 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,812 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,812 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,812 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,816 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,816 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,818 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,818 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,818 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,819 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,820 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,825 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,826 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,828 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,828 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ec002e2b421a4e468df41baeed37731c 2023-03-28 20:55:38,928 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:38,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:38,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:38,931 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:38,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:38,931 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:38,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,934 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:38,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:38,937 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,937 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:38,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740e3ce2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:38,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:38,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:38,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:38,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:38,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:38,947 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:38,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:38,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:38,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:38,947 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:38,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:38,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:38,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:38,951 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:38,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:38,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:38,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:38,953 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:38,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:38,954 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:38,954 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:38,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:38,954 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:38,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:38,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:38,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,054 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,057 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:39,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152aa998[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,068 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,068 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,072 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,072 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,073 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,073 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,082 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,082 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:39,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680036999154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:39,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680036999159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:39,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680036999161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:39,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680036999162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:39,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680036999163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,182 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,186 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,188 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,192 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312105ec[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,194 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,194 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,197 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,197 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,202 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,202 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,204 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,207 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,210 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:39,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,211 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,211 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,211 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,211 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,212 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,212 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f6a6b2fb67c24288ae4b388ae93c10cc 2023-03-28 20:55:39,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/5d87f8808ad1461ba791eb1a6807d187 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187 2023-03-28 20:55:39,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187, entries=150, sequenceid=12, filesize=11.6 K 2023-03-28 20:55:39,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ec002e2b421a4e468df41baeed37731c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c 2023-03-28 20:55:39,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c, entries=150, sequenceid=12, filesize=11.6 K 2023-03-28 20:55:39,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f6a6b2fb67c24288ae4b388ae93c10cc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc 2023-03-28 20:55:39,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc, entries=150, sequenceid=12, filesize=11.6 K 2023-03-28 20:55:39,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 688ef3a4c2e822fe7803c6aa582a3052 in 1264ms, sequenceid=12, compaction requested=false 2023-03-28 20:55:39,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,312 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,315 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,320 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-28 20:55:39,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:39,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:39,321 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:39,321 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:39,321 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:39,321 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:39,328 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/dcbc18c3caf44d8fbcf794b013adbf2d 2023-03-28 20:55:39,336 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/f818b0e3f60b423cb8164c5f5bc83790 2023-03-28 20:55:39,345 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a52ec68eecf54b68bb291a05fce75c00 2023-03-28 20:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/dcbc18c3caf44d8fbcf794b013adbf2d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d 2023-03-28 20:55:39,352 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d, entries=150, sequenceid=49, filesize=11.6 K 2023-03-28 20:55:39,352 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/f818b0e3f60b423cb8164c5f5bc83790 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790 2023-03-28 20:55:39,355 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790, entries=150, sequenceid=49, filesize=11.6 K 2023-03-28 20:55:39,356 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a52ec68eecf54b68bb291a05fce75c00 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00 2023-03-28 20:55:39,359 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00, entries=150, sequenceid=49, filesize=11.6 K 2023-03-28 20:55:39,360 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 688ef3a4c2e822fe7803c6aa582a3052 in 40ms, sequenceid=49, compaction requested=false 2023-03-28 20:55:39,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,360 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,362 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,362 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512664ad[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,367 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,367 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,367 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,377 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,377 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,378 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,383 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,383 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,383 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,383 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,383 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,383 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,483 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,486 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,489 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,489 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,492 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,492 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,492 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,492 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,493 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,493 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54caab47[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,498 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,502 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,502 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,504 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,506 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,508 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,508 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,509 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,509 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,509 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,509 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,509 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,609 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,612 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,617 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,619 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,621 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146cae2f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,625 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,625 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,625 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,625 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,627 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,629 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,629 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,633 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,635 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,635 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,635 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,635 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,735 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,748 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,749 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,749 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,750 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,756 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5bd72b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,760 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,760 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,761 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,765 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,765 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,767 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,768 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,774 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,774 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,874 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:39,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:39,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:39,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:39,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:39,877 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:39,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:39,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:39,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:39,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:39,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:39,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:39,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:39,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:39,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:39,881 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:39,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,881 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:39,882 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:39,882 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:39,883 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:39,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3acd21[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:39,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:39,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:39,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:39,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:39,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:39,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:39,894 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:39,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:39,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:39,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:39,894 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:39,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:39,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:39,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:39,896 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:39,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:39,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:39,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:39,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:39,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:39,901 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:39,901 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:39,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:39,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,001 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,005 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,006 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,008 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:40,014 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,015 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eabd712[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:40,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:40,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,017 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:40,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:40,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:40,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:40,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,019 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,019 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,020 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:40,022 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:40,022 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:40,022 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:40,022 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:40,022 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:40,024 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,024 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,024 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,026 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:40,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:40,029 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:40,029 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:40,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,129 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,141 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,141 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,141 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,141 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,142 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,145 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,146 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,149 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,149 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,149 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:40,149 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:40,149 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:40,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,153 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,153 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:40,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f47053e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:40,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:40,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:40,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:40,165 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:40,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:40,167 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:40,172 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:40,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:40,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:40,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:40,172 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:40,178 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,178 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:40,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:40,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,184 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,184 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,184 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:40,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,185 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,185 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:40,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:40,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,185 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:40,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/40f3f2d0bdb144e69ea4d6a8758cfb0d 2023-03-28 20:55:40,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036940195, max timestamp=1680036940212Num uniques 859; 2023-03-28 20:55:40,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:40,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036940195, max timestamp=1680036940212Num uniques 859; 2023-03-28 20:55:40,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:40,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036940195, max timestamp=1680036940212Num uniques 859; 2023-03-28 20:55:40,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:40,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/b70e3ae7447f423b81330834f46d496b 2023-03-28 20:55:40,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037000233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037000234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037000240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037000244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037000245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,286 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,288 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,289 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,290 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,290 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:40,292 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:40,293 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:40,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd66f72[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:40,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:40,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,298 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:40,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:40,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:40,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:40,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:40,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:40,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:40,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:40,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:40,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:40,303 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,303 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,303 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,305 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:40,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,307 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:40,307 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037000346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037000346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037000347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037000348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037000355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,407 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,410 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,413 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:40,416 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:40,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559cf00c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:40,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:40,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,419 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:40,419 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,420 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:40,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:40,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:40,428 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:40,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:40,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:40,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:40,428 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:40,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:40,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:40,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,435 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:40,435 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:40,435 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,535 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,538 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,540 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:40,542 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:40,542 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:40,545 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:40,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451faeea[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:40,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:40,546 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,549 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:40,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:40,552 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:40,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:40,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:40,552 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:40,552 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:40,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,554 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,554 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:40,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037000555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,560 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:40,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:40,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:40,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,560 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:40,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:40,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037000556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037000556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037000558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,562 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:40,562 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:40,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037000562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/0d2c3a698b564a42819d69fe80b9d5b7 2023-03-28 20:55:40,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/40f3f2d0bdb144e69ea4d6a8758cfb0d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d 2023-03-28 20:55:40,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d, entries=200, sequenceid=66, filesize=13.9 K 2023-03-28 20:55:40,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/b70e3ae7447f423b81330834f46d496b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b 2023-03-28 20:55:40,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b, entries=150, sequenceid=66, filesize=11.6 K 2023-03-28 20:55:40,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/0d2c3a698b564a42819d69fe80b9d5b7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7 2023-03-28 20:55:40,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7, entries=150, sequenceid=66, filesize=11.6 K 2023-03-28 20:55:40,662 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:40,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 688ef3a4c2e822fe7803c6aa582a3052 in 483ms, sequenceid=66, compaction requested=true 2023-03-28 20:55:40,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:40,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:40,662 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:40,663 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:40,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:40,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:40,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:40,663 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:40,664 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:40,664 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,664 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=37.1 K 2023-03-28 20:55:40,664 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:40,664 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:40,664 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,664 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=34.8 K 2023-03-28 20:55:40,664 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5d87f8808ad1461ba791eb1a6807d187, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680036937978 2023-03-28 20:55:40,665 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ec002e2b421a4e468df41baeed37731c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680036937978 2023-03-28 20:55:40,665 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting dcbc18c3caf44d8fbcf794b013adbf2d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680036938036 2023-03-28 20:55:40,665 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f818b0e3f60b423cb8164c5f5bc83790, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680036938036 2023-03-28 20:55:40,666 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 40f3f2d0bdb144e69ea4d6a8758cfb0d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940173 2023-03-28 20:55:40,666 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b70e3ae7447f423b81330834f46d496b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940175 2023-03-28 20:55:40,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:40,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:40,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:40,668 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:40,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:40,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:40,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,670 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,670 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:40,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:40,671 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,671 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:40,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:40,672 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:40,673 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:40,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:40,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:40,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:40,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:40,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:40,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:40,678 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,678 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:40,678 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-28 20:55:40,678 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:40,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:40,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:40,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:40,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:40,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:40,700 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/63a54e3b809b49aa9e8f9c4ead8b7e04 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/63a54e3b809b49aa9e8f9c4ead8b7e04 2023-03-28 20:55:40,702 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ff9fefe3d1a745d381976d45d7611d3d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ff9fefe3d1a745d381976d45d7611d3d 2023-03-28 20:55:40,705 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 63a54e3b809b49aa9e8f9c4ead8b7e04(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:40,705 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:40,705 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036940662; duration=0sec 2023-03-28 20:55:40,705 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:40,705 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:40,705 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/d4a31391cb8f41edb966796f15c10ac3 2023-03-28 20:55:40,707 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:40,707 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:40,707 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:40,707 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=34.8 K 2023-03-28 20:55:40,708 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f6a6b2fb67c24288ae4b388ae93c10cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680036937978 2023-03-28 20:55:40,708 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into ff9fefe3d1a745d381976d45d7611d3d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:40,709 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:40,709 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036940662; duration=0sec 2023-03-28 20:55:40,709 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:40,709 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a52ec68eecf54b68bb291a05fce75c00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680036938036 2023-03-28 20:55:40,710 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0d2c3a698b564a42819d69fe80b9d5b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940175 2023-03-28 20:55:40,716 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:40,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:40,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:40,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037000871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037000892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037000892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037000894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037000894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:40,954 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:55:40,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:40,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037000994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037001000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037001001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037001001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037001001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,133 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f686a50705c14dc8871bc93fc4d3f9a5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f686a50705c14dc8871bc93fc4d3f9a5 2023-03-28 20:55:41,136 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7488c82f34f047889dc98a4ac02d7f7a 2023-03-28 20:55:41,137 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into f686a50705c14dc8871bc93fc4d3f9a5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:41,137 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:41,137 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=13, startTime=1680036940663; duration=0sec 2023-03-28 20:55:41,138 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:41,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037001199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037001204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037001205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037001205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037001205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037001502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037001509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037001510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037001510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:41,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037001511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,545 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/6d04cd31b56c46aeafe433f148f7f929 2023-03-28 20:55:41,549 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/d4a31391cb8f41edb966796f15c10ac3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3 2023-03-28 20:55:41,552 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3, entries=150, sequenceid=100, filesize=11.6 K 2023-03-28 20:55:41,553 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7488c82f34f047889dc98a4ac02d7f7a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a 2023-03-28 20:55:41,556 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a, entries=150, sequenceid=100, filesize=11.6 K 2023-03-28 20:55:41,557 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/6d04cd31b56c46aeafe433f148f7f929 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929 2023-03-28 20:55:41,560 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929, entries=150, sequenceid=100, filesize=11.6 K 2023-03-28 20:55:41,560 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 688ef3a4c2e822fe7803c6aa582a3052 in 882ms, sequenceid=100, compaction requested=false 2023-03-28 20:55:41,560 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:41,560 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:41,560 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:41,561 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:41,562 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:41,562 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,563 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:41,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:41,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42762d82[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:41,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:41,564 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,565 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:41,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:41,565 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:41,566 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,566 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:41,568 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:41,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:41,570 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:41,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:41,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:41,570 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:41,570 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:41,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:41,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,572 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:41,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:41,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,574 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:41,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:41,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:41,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:41,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,577 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:41,577 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:41,677 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:41,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:41,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:41,680 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:41,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:41,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:41,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:41,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:41,682 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,682 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:41,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:41,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:41,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:41,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:41,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:41,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:41,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:41,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:41,684 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:55:41,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:41,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:41,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:41,685 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:41,685 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:41,685 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:41,691 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/46efe59250d6471eb97d03b92acb41e0 2023-03-28 20:55:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:42,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:42,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036942051, max timestamp=1680036942062Num uniques 859; 2023-03-28 20:55:42,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:42,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036942051, max timestamp=1680036942062Num uniques 859; 2023-03-28 20:55:42,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:42,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036942051, max timestamp=1680036942062Num uniques 859; 2023-03-28 20:55:42,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:42,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037002080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037002083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037002084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037002085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037002085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,099 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/f849b4836f90417d863287d5581623cf 2023-03-28 20:55:42,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037002186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037002188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037002192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037002192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037002193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037002393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037002393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037002398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037002399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037002400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,507 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/20c54ce005ea4d0b92e355716c80a22a 2023-03-28 20:55:42,511 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/46efe59250d6471eb97d03b92acb41e0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0 2023-03-28 20:55:42,514 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0, entries=150, sequenceid=115, filesize=11.6 K 2023-03-28 20:55:42,515 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/f849b4836f90417d863287d5581623cf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf 2023-03-28 20:55:42,518 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf, entries=150, sequenceid=115, filesize=11.6 K 2023-03-28 20:55:42,518 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/20c54ce005ea4d0b92e355716c80a22a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a 2023-03-28 20:55:42,521 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a, entries=150, sequenceid=115, filesize=11.6 K 2023-03-28 20:55:42,522 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 688ef3a4c2e822fe7803c6aa582a3052 in 838ms, sequenceid=115, compaction requested=true 2023-03-28 20:55:42,522 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:42,522 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,522 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:42,522 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:42,522 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,522 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:42,522 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:42,525 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,525 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:42,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d246741[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:42,527 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:42,527 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,528 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:42,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:42,530 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:42,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:42,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:42,532 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:42,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:42,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:42,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:42,532 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:42,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,538 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,539 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:42,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:42,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,543 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:42,543 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:42,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,643 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:42,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:42,646 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:42,646 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:42,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:42,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,648 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:42,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:42,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:42,650 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:42,650 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:42,651 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:42,651 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:42,657 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/fefbe3d290854347b6876ffb306e009f 2023-03-28 20:55:42,666 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bf9fb63f2f02417bb99a96e63feae29e 2023-03-28 20:55:42,683 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/0ecb9c8d8db5484a8f1e9e68a0824872 2023-03-28 20:55:42,687 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/fefbe3d290854347b6876ffb306e009f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f 2023-03-28 20:55:42,690 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f, entries=150, sequenceid=149, filesize=11.8 K 2023-03-28 20:55:42,691 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bf9fb63f2f02417bb99a96e63feae29e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e 2023-03-28 20:55:42,695 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e, entries=150, sequenceid=149, filesize=11.8 K 2023-03-28 20:55:42,696 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/0ecb9c8d8db5484a8f1e9e68a0824872 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872 2023-03-28 20:55:42,699 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872, entries=150, sequenceid=149, filesize=11.8 K 2023-03-28 20:55:42,700 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for 688ef3a4c2e822fe7803c6aa582a3052 in 50ms, sequenceid=149, compaction requested=true 2023-03-28 20:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:42,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:42,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8aec79[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:42,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:42,706 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:42,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:42,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:42,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:42,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:42,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,710 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:42,710 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:42,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:42,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:42,713 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:42,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:42,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:42,713 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:42,713 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:42,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,715 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,716 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,719 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:42,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:42,722 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:42,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,722 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036942726, max timestamp=1680036942742Num uniques 859; 2023-03-28 20:55:42,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:42,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036942726, max timestamp=1680036942742Num uniques 859; 2023-03-28 20:55:42,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:42,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036942726, max timestamp=1680036942742Num uniques 859; 2023-03-28 20:55:42,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:42,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037002763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037002767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037002770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037002770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037002771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,822 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:42,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:42,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:42,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:42,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:42,826 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:42,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:42,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:42,826 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:42,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:42,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:42,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:42,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:42,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:42,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:42,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:42,842 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:42,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37da6b9c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:42,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:42,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,847 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:42,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:42,847 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:42,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:42,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:42,856 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:42,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:42,856 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:42,858 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,858 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,858 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,860 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:42,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:42,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:42,863 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:42,863 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037002872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037002877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037002877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037002885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037002886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,963 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:42,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:42,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:42,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:42,966 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:42,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:42,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:42,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:42,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:42,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:42,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:42,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:42,970 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:42,970 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:42,973 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b35ad8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:42,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:42,975 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,976 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:42,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:42,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:42,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:42,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:42,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:42,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:42,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:42,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:42,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:42,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:42,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:42,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,990 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:42,990 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:42,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:42,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:42,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:42,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:42,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:42,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:42,991 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:42,992 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,992 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:42,992 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:42,992 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:42,992 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037003084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037003085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037003086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,092 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,095 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,096 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,099 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,099 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,099 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,099 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,099 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab33360[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,103 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,104 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,104 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037003100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037003101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,109 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,109 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,115 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,118 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,118 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:43,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/0a0e11973506431c85d68873edab5007 2023-03-28 20:55:43,218 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,222 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,224 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,225 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,225 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,225 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,225 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,228 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282e56b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,229 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,229 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,231 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,232 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,234 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,234 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,242 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,242 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,342 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,346 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,347 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,347 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,349 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,350 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,351 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6660e82c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,355 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,358 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,364 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,364 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,366 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,373 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,373 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,373 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,373 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037003392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037003393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037003393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037003409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037003409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,473 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,480 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,480 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,482 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,483 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,483 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,484 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385b1ba2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,489 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,495 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,495 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,499 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,503 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:43,506 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,506 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,506 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,506 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bf46d4d981e44b19a3008ab59dbe1e44 2023-03-28 20:55:43,606 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,609 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,613 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,613 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,613 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,613 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,614 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3007fa[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,618 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,620 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,621 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,621 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,622 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,624 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,624 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,626 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,627 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,627 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,632 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,632 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,633 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,633 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:43,733 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,746 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,750 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,751 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,756 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,758 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8c1c4a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,762 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,766 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,766 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,766 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,771 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,772 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,776 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,776 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:43,777 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,877 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:43,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:43,880 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:43,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:43,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:43,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,882 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:43,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:43,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:43,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:43,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:43,883 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:43,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,889 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fccb83f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:43,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:43,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:43,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,897 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:43,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:43,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:43,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:43,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:43,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:43,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:43,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:43,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:43,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,904 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,907 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:43,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:43,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:43,907 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:43,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:43,907 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:43,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:43,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037003903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037003904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037003910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037003921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037003925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:43,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/bf6b45206c524ab1a8de309dbc0841b9 2023-03-28 20:55:43,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/0a0e11973506431c85d68873edab5007 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007 2023-03-28 20:55:43,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007, entries=300, sequenceid=159, filesize=18.8 K 2023-03-28 20:55:43,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bf46d4d981e44b19a3008ab59dbe1e44 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44 2023-03-28 20:55:43,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44, entries=150, sequenceid=159, filesize=11.8 K 2023-03-28 20:55:43,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/bf6b45206c524ab1a8de309dbc0841b9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9 2023-03-28 20:55:43,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9, entries=150, sequenceid=159, filesize=11.8 K 2023-03-28 20:55:43,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 688ef3a4c2e822fe7803c6aa582a3052 in 1243ms, sequenceid=159, compaction requested=true 2023-03-28 20:55:43,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:43,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:43,954 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:55:43,954 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:55:43,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:43,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 67003 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:55:43,956 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:43,956 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/63a54e3b809b49aa9e8f9c4ead8b7e04, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=65.4 K 2023-03-28 20:55:43,956 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:43,956 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ff9fefe3d1a745d381976d45d7611d3d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=58.4 K 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 63a54e3b809b49aa9e8f9c4ead8b7e04, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940175 2023-03-28 20:55:43,956 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ff9fefe3d1a745d381976d45d7611d3d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940175 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d4a31391cb8f41edb966796f15c10ac3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680036940220 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7488c82f34f047889dc98a4ac02d7f7a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680036940220 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 46efe59250d6471eb97d03b92acb41e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680036940865 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f849b4836f90417d863287d5581623cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680036940865 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fefbe3d290854347b6876ffb306e009f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680036942079 2023-03-28 20:55:43,957 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bf9fb63f2f02417bb99a96e63feae29e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680036942079 2023-03-28 20:55:43,958 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0a0e11973506431c85d68873edab5007, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680036942702 2023-03-28 20:55:43,958 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bf46d4d981e44b19a3008ab59dbe1e44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680036942704 2023-03-28 20:55:43,967 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:43,970 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:44,007 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,010 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,012 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,013 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,014 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-28 20:55:44,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:44,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:44,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:44,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:44,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:44,015 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:44,025 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/19b081ea322749a3a9cdd6b26d3f99c3 2023-03-28 20:55:44,037 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/23155006531a41da84fc07b25e744f4a 2023-03-28 20:55:44,047 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a81cae4acd27429ca8c7e4f041641cb2 2023-03-28 20:55:44,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/19b081ea322749a3a9cdd6b26d3f99c3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3 2023-03-28 20:55:44,054 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3, entries=150, sequenceid=194, filesize=11.8 K 2023-03-28 20:55:44,055 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/23155006531a41da84fc07b25e744f4a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a 2023-03-28 20:55:44,057 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a, entries=150, sequenceid=194, filesize=11.8 K 2023-03-28 20:55:44,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a81cae4acd27429ca8c7e4f041641cb2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2 2023-03-28 20:55:44,066 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2, entries=150, sequenceid=194, filesize=11.8 K 2023-03-28 20:55:44,067 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 688ef3a4c2e822fe7803c6aa582a3052 in 53ms, sequenceid=194, compaction requested=true 2023-03-28 20:55:44,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6faa346f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,073 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,074 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,076 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,080 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,080 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,082 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,085 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,085 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:44,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,086 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,086 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,086 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,187 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,192 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,195 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145ef1b7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,209 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,209 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,210 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,211 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,212 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,212 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,216 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,216 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,221 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,221 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,224 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:44,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,231 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,231 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,331 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,334 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,335 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,336 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,336 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,337 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,337 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,338 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,338 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,338 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,338 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,339 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,340 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3ed13a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,342 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,342 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,350 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,353 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,353 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,353 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,354 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,355 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,355 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,359 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,359 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,359 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,380 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ee41338741944b9a8ff9caa60a28d3db as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ee41338741944b9a8ff9caa60a28d3db 2023-03-28 20:55:44,384 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into ee41338741944b9a8ff9caa60a28d3db(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:44,384 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,384 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=11, startTime=1680036943954; duration=0sec 2023-03-28 20:55:44,384 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:44,384 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-28 20:55:44,385 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71869 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-28 20:55:44,386 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:44,386 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,386 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f686a50705c14dc8871bc93fc4d3f9a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=70.2 K 2023-03-28 20:55:44,386 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f686a50705c14dc8871bc93fc4d3f9a5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680036940175 2023-03-28 20:55:44,386 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6d04cd31b56c46aeafe433f148f7f929, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680036940220 2023-03-28 20:55:44,387 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 20c54ce005ea4d0b92e355716c80a22a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680036940865 2023-03-28 20:55:44,387 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0ecb9c8d8db5484a8f1e9e68a0824872, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680036942079 2023-03-28 20:55:44,388 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bf6b45206c524ab1a8de309dbc0841b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680036942704 2023-03-28 20:55:44,388 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/2357018e6e204a0c8862b48199b78fc2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/2357018e6e204a0c8862b48199b78fc2 2023-03-28 20:55:44,388 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a81cae4acd27429ca8c7e4f041641cb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680036942762 2023-03-28 20:55:44,392 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 2357018e6e204a0c8862b48199b78fc2(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:44,392 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,392 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=11, startTime=1680036943954; duration=0sec 2023-03-28 20:55:44,392 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:44,398 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#474 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:44,459 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,462 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,463 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,463 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,465 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,466 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,466 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,466 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,467 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,469 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70599f02[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,472 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,473 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,475 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,476 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,481 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,481 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,481 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,484 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,486 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,486 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,486 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,586 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,590 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,593 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,596 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,596 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,596 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,596 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,598 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eeed42a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,601 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,601 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,603 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,605 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,611 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,611 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,613 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,613 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,614 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,614 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,616 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:44,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,627 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,627 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,727 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,744 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,747 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,747 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,754 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dde5709[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,756 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,759 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,759 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,763 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,763 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,769 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,770 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,774 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,774 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,774 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:44,806 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/32296f6771e64d60bce42bcd42bb6a84 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/32296f6771e64d60bce42bcd42bb6a84 2023-03-28 20:55:44,810 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into 32296f6771e64d60bce42bcd42bb6a84(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:44,810 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,810 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=10, startTime=1680036943954; duration=0sec 2023-03-28 20:55:44,810 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:44,874 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:44,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:44,878 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:44,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:44,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:44,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,879 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,880 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:44,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:44,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4185e489[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:44,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:44,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:44,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:44,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:44,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:44,895 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:44,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:44,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:44,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:44,895 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:44,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,897 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:44,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:44,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:44,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:44,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:44,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:44,902 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:44,902 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:44,902 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:44,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:44,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037004948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037004952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036944937, max timestamp=1680036944955Num uniques 859; 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036944937, max timestamp=1680036944955Num uniques 859; 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036944937, max timestamp=1680036944955Num uniques 859; 2023-03-28 20:55:44,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:45,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037004990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,002 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037004991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037004991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,015 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,018 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,033 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,033 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,034 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,034 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d98d234[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,038 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,038 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,040 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,042 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,045 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,045 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,050 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,050 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,050 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,060 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,060 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,060 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037005057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037005057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037005104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037005105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037005105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,160 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,163 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,163 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,167 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,168 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456f1629[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,170 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,172 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,174 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,176 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,176 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,176 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,177 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,178 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,190 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,190 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,190 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037005265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037005265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,290 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,293 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,293 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,297 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,297 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,297 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,297 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad67792[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,306 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,306 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,308 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,308 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,311 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,312 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,312 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037005313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037005313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037005314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/6a98e11ec6ef49818206bea2b78c66f3 2023-03-28 20:55:45,412 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,415 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,417 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,419 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,422 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749979ed[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,425 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,426 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,426 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,427 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,427 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,427 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,427 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,429 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,429 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,429 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,436 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,436 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,437 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,437 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,437 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,437 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,437 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,537 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,540 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,541 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,541 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,543 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,543 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,543 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,543 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,545 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2b1c38[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,548 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,550 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,553 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,553 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,555 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,561 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,562 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,562 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,562 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037005568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037005569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037005617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037005620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:45,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037005623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,662 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,665 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,669 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,673 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724895aa[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,676 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,678 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,681 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,681 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,688 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,692 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,692 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,693 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ecdeb5f818be43fbbfd525cb688ab8ee 2023-03-28 20:55:45,793 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,796 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,796 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,798 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,798 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8b0696[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,805 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,806 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,807 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,809 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,810 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,811 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,811 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,811 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,812 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,816 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,816 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:45,816 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,817 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,917 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:45,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:45,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:45,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:45,920 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:45,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:45,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:45,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,922 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,922 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:45,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:45,924 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,924 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:45,924 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:45,924 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:45,925 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,925 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e24a1b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:45,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:45,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:45,929 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,929 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:45,931 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:45,933 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:45,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:45,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:45,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:45,933 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:45,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:45,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,936 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:45,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:45,936 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:45,939 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:45,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:45,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:45,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,940 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:45,940 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:45,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,040 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,043 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,045 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,046 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,047 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,047 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,047 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:46,047 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,055 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733f5063[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,058 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,058 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,062 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,062 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,069 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,069 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,070 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037006073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037006073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037006123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037006124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037006127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/70f4b646271d48a08c0ebd3a85001035 2023-03-28 20:55:46,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/6a98e11ec6ef49818206bea2b78c66f3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3 2023-03-28 20:55:46,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3, entries=200, sequenceid=214, filesize=14.1 K 2023-03-28 20:55:46,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ecdeb5f818be43fbbfd525cb688ab8ee as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee 2023-03-28 20:55:46,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee, entries=150, sequenceid=214, filesize=11.8 K 2023-03-28 20:55:46,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/70f4b646271d48a08c0ebd3a85001035 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035 2023-03-28 20:55:46,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035, entries=150, sequenceid=214, filesize=11.8 K 2023-03-28 20:55:46,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 688ef3a4c2e822fe7803c6aa582a3052 in 1241ms, sequenceid=214, compaction requested=true 2023-03-28 20:55:46,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:46,168 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:46,168 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:46,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:46,168 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:46,169 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38772 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:46,169 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:46,169 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,169 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/2357018e6e204a0c8862b48199b78fc2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=37.9 K 2023-03-28 20:55:46,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:46,169 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,170 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:46,170 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,170 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ee41338741944b9a8ff9caa60a28d3db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=35.5 K 2023-03-28 20:55:46,170 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 2357018e6e204a0c8862b48199b78fc2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680036942704 2023-03-28 20:55:46,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,170 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ee41338741944b9a8ff9caa60a28d3db, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680036942704 2023-03-28 20:55:46,170 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 19b081ea322749a3a9cdd6b26d3f99c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680036942762 2023-03-28 20:55:46,171 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 23155006531a41da84fc07b25e744f4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680036942762 2023-03-28 20:55:46,171 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 6a98e11ec6ef49818206bea2b78c66f3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680036944923 2023-03-28 20:55:46,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,172 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ecdeb5f818be43fbbfd525cb688ab8ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680036944923 2023-03-28 20:55:46,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,173 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,175 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,177 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:46,177 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:46,179 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:46,180 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:46,192 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/c808a597082648bca43913daf81946c3 2023-03-28 20:55:46,200 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cd82ed9ac8bb422794f0eedf469f1eca 2023-03-28 20:55:46,207 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/c3b4b72e4fd7479b8bad81b24f893438 2023-03-28 20:55:46,210 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/c808a597082648bca43913daf81946c3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3 2023-03-28 20:55:46,212 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3, entries=150, sequenceid=248, filesize=11.8 K 2023-03-28 20:55:46,213 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cd82ed9ac8bb422794f0eedf469f1eca as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca 2023-03-28 20:55:46,216 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca, entries=150, sequenceid=248, filesize=11.8 K 2023-03-28 20:55:46,217 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/c3b4b72e4fd7479b8bad81b24f893438 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438 2023-03-28 20:55:46,219 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438, entries=150, sequenceid=248, filesize=11.8 K 2023-03-28 20:55:46,220 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 688ef3a4c2e822fe7803c6aa582a3052 in 43ms, sequenceid=248, compaction requested=true 2023-03-28 20:55:46,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,222 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456c7e6b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,225 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,225 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,226 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,229 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,233 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,233 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,237 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:46,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,240 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,240 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,340 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,344 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,345 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,345 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,350 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d06fe6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,352 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,352 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,353 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,355 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,358 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,358 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:46,366 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,366 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,466 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,470 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,470 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,480 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,481 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,481 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,481 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301457b5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,490 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,494 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,494 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,500 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,500 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,500 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,590 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/c09b488a97c943c4b565f48e4be842d0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/c09b488a97c943c4b565f48e4be842d0 2023-03-28 20:55:46,593 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/5d37565d1f6342e3bde4c019728e18a5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d37565d1f6342e3bde4c019728e18a5 2023-03-28 20:55:46,594 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into c09b488a97c943c4b565f48e4be842d0(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:46,594 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,594 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036946168; duration=0sec 2023-03-28 20:55:46,595 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:46,595 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:46,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:46,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:46,596 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,596 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/32296f6771e64d60bce42bcd42bb6a84, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=35.6 K 2023-03-28 20:55:46,596 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 32296f6771e64d60bce42bcd42bb6a84, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680036942762 2023-03-28 20:55:46,597 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 5d37565d1f6342e3bde4c019728e18a5(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:46,597 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,597 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036946168; duration=0sec 2023-03-28 20:55:46,597 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 70f4b646271d48a08c0ebd3a85001035, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680036944923 2023-03-28 20:55:46,597 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:46,598 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c3b4b72e4fd7479b8bad81b24f893438, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680036944981 2023-03-28 20:55:46,601 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,603 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:46,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,606 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,609 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,609 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,613 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a5512eb4f15c4f06aa7c8c8b553be53e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a5512eb4f15c4f06aa7c8c8b553be53e 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,617 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into a5512eb4f15c4f06aa7c8c8b553be53e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,617 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,617 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,617 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=13, startTime=1680036946168; duration=0sec 2023-03-28 20:55:46,618 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:46,619 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,619 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3921b59c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,622 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,623 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,625 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,628 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,628 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,629 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,629 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,631 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,631 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,633 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,637 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,637 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,638 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,638 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,640 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,640 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,640 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,640 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,641 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,738 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,742 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,743 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,743 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,748 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,748 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,749 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,749 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,750 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f180bf5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,752 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,752 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,754 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,756 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,760 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,760 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,760 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,763 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,765 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,768 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,769 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,769 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,769 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,869 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:46,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:46,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:46,872 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:46,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:46,872 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:46,873 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,873 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,875 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,875 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:46,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:46,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:46,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:46,876 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:46,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:46,877 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:46,877 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,877 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:46,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,879 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:46,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af2e8af[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:46,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:46,881 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,882 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:46,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:46,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:46,894 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:46,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:46,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:46,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:46,894 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:46,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:46,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,897 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:46,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:46,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:46,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:46,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:46,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:46,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:46,901 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,901 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:46,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,001 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,005 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,007 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,007 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,009 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,009 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,013 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4849a5bb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,016 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,021 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,021 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,021 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,022 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,023 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,024 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,025 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:47,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,028 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,028 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,028 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:47,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:47,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:47,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036947110, max timestamp=1680036947127Num uniques 859; 2023-03-28 20:55:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036947110, max timestamp=1680036947127Num uniques 859; 2023-03-28 20:55:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:47,129 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036947110, max timestamp=1680036947127Num uniques 859; 2023-03-28 20:55:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,131 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,131 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,132 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,132 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,132 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,132 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,135 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf0647b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,138 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,140 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,142 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,144 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,144 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,146 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,147 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,150 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,151 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,151 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,151 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:47,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037007149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037007150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037007150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037007151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037007157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,251 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,254 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,254 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,258 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,258 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,258 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,258 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037007258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037007258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037007259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037007261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037007261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,266 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e89fb3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,269 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,269 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,269 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,273 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,273 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,282 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,282 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,282 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,282 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,282 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,283 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,283 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,283 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,283 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,283 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,283 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,383 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,386 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,388 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,388 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,389 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,389 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,390 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,390 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,390 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,390 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,392 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e12acf1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,394 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,394 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,396 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,396 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,396 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,399 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,402 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,402 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,403 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,403 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,404 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,407 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,410 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,410 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:47,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,409 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,410 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,410 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037007463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037007463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037007465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037007466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037007466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/3da61ebc9a344eccab3ee58f61b832cc 2023-03-28 20:55:47,511 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,515 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,515 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,517 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,517 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,519 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,519 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,519 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,519 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,526 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf3a8f5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,528 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,530 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,535 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,535 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,536 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,538 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,541 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,541 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,541 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,542 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,542 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,542 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,548 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,642 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,648 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,648 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,649 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,650 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,650 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,652 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,653 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,653 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,658 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,658 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,658 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,658 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,660 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145fca4c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,665 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,666 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,669 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,672 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,672 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,672 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,672 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,672 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,677 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,679 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,688 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,688 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,688 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:47,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,688 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037007764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037007766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037007773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037007773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037007774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,789 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,792 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,792 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,797 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,797 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,797 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,797 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73adda73[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,801 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,803 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,805 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,807 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,808 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,809 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,809 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,809 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,810 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,814 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,814 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,814 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,814 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/97bd70a337ae47e9a04833dc3560eacc 2023-03-28 20:55:47,914 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:47,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:47,917 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:47,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:47,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:47,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:47,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:47,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:47,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:47,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:47,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:47,923 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,923 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a395d3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:47,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:47,925 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:47,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,930 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:47,933 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:47,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:47,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:47,933 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:47,933 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:47,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:47,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:47,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,942 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,946 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:47,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:47,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:47,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:47,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:47,946 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:47,946 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:47,947 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,947 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:47,947 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,047 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:48,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:48,050 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:48,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:48,050 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:48,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,051 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,052 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:48,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:48,053 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,054 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:48,054 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:48,054 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:48,055 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee13032[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:48,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:48,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,058 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:48,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:48,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,059 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:48,062 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,062 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:48,065 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:48,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:48,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:48,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:48,065 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:48,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,068 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,071 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:48,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,071 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:48,071 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:48,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,171 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:48,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:48,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:48,175 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:48,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:48,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:48,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,175 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,180 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,181 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:48,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:48,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:48,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,181 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:48,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:48,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:48,182 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:48,184 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,184 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14069808[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:48,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:48,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,187 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:48,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:48,189 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:48,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:48,192 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:48,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:48,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:48,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:48,192 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:48,194 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,194 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,194 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,196 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:48,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,199 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:48,200 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:48,199 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,200 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037008272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037008275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037008278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037008281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:48,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037008282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,300 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:48,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:48,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:48,302 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:48,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:48,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:48,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:48,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:48,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:48,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:48,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:48,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:48,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:48,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:48,306 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,306 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:48,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,309 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,310 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e5b6b9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:48,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:48,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:48,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:48,313 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:48,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:48,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:48,316 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:48,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:48,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:48,316 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:48,316 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:48,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f657eeaeab924bcc8422eff1b1e16649 2023-03-28 20:55:48,319 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:48,320 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,320 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:48,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/3da61ebc9a344eccab3ee58f61b832cc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc 2023-03-28 20:55:48,320 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,322 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc, entries=100, sequenceid=267, filesize=9.5 K 2023-03-28 20:55:48,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/97bd70a337ae47e9a04833dc3560eacc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:48,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:48,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc, entries=100, sequenceid=267, filesize=9.5 K 2023-03-28 20:55:48,327 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:48,327 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:48,327 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:48,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f657eeaeab924bcc8422eff1b1e16649 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649 2023-03-28 20:55:48,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649, entries=100, sequenceid=267, filesize=9.5 K 2023-03-28 20:55:48,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 688ef3a4c2e822fe7803c6aa582a3052 in 1243ms, sequenceid=267, compaction requested=true 2023-03-28 20:55:48,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:48,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:48,332 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:48,332 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:48,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:48,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:48,333 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,333 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,333 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d37565d1f6342e3bde4c019728e18a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=33.4 K 2023-03-28 20:55:48,333 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/c09b488a97c943c4b565f48e4be842d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=33.4 K 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5d37565d1f6342e3bde4c019728e18a5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680036944923 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c09b488a97c943c4b565f48e4be842d0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680036944923 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c808a597082648bca43913daf81946c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680036944981 2023-03-28 20:55:48,333 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cd82ed9ac8bb422794f0eedf469f1eca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680036944981 2023-03-28 20:55:48,334 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 3da61ebc9a344eccab3ee58f61b832cc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680036947087 2023-03-28 20:55:48,334 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 97bd70a337ae47e9a04833dc3560eacc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680036947087 2023-03-28 20:55:48,347 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:48,347 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:48,358 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/0963c25c2b3647deaab1bc2c9868ee90 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/0963c25c2b3647deaab1bc2c9868ee90 2023-03-28 20:55:48,362 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into 0963c25c2b3647deaab1bc2c9868ee90(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:48,362 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:48,362 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036948332; duration=0sec 2023-03-28 20:55:48,362 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:48,362 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-28 20:55:48,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-28 20:55:48,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-28 20:55:48,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. because compaction request was cancelled 2023-03-28 20:55:48,427 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:48,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:48,430 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:48,430 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:48,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:48,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,432 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:48,432 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:48,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:48,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:48,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:48,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:48,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:48,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:48,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:48,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:48,434 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:48,434 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:48,435 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:48,435 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:48,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/526745725c30427080f4c660c35eb4d6 2023-03-28 20:55:48,759 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/4feb7fe34b1d48cd87548289446b1baa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/4feb7fe34b1d48cd87548289446b1baa 2023-03-28 20:55:48,762 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 4feb7fe34b1d48cd87548289446b1baa(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:48,763 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:48,763 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036948332; duration=0sec 2023-03-28 20:55:48,763 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:48,858 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/1ac2080198b746649bab464774444273 2023-03-28 20:55:49,268 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/e0501ecdee4c464482e40cf77ca5bdd0 2023-03-28 20:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/526745725c30427080f4c660c35eb4d6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6 2023-03-28 20:55:49,277 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6, entries=150, sequenceid=304, filesize=11.9 K 2023-03-28 20:55:49,278 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/1ac2080198b746649bab464774444273 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273 2023-03-28 20:55:49,285 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273, entries=150, sequenceid=304, filesize=11.9 K 2023-03-28 20:55:49,285 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/e0501ecdee4c464482e40cf77ca5bdd0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0 2023-03-28 20:55:49,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:49,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:49,294 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0, entries=150, sequenceid=304, filesize=11.9 K 2023-03-28 20:55:49,295 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=33.54 KB/34350 for 688ef3a4c2e822fe7803c6aa582a3052 in 861ms, sequenceid=304, compaction requested=true 2023-03-28 20:55:49,295 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:49,295 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:49,297 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:49,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:49,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d7cb75b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:49,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:49,305 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/cd034fad1d714b8491ca04cf78ce5d73 2023-03-28 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:49,310 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:49,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:49,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:49,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:49,310 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:49,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,318 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:49,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:49,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,319 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,319 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:49,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037009321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036949316, max timestamp=1680036949327Num uniques 859; 2023-03-28 20:55:49,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036949316, max timestamp=1680036949327Num uniques 859; 2023-03-28 20:55:49,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:49,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680036949316, max timestamp=1680036949327Num uniques 859; 2023-03-28 20:55:49,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:49,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-28 20:55:49,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/d7d9f9ccd77a46b98ad6778b5eca5d9c 2023-03-28 20:55:49,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f 2023-03-28 20:55:49,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/cd034fad1d714b8491ca04cf78ce5d73 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73 2023-03-28 20:55:49,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73, entries=200, sequenceid=315, filesize=14.3 K 2023-03-28 20:55:49,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/d7d9f9ccd77a46b98ad6778b5eca5d9c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c 2023-03-28 20:55:49,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037009356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037009365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c, entries=150, sequenceid=315, filesize=11.9 K 2023-03-28 20:55:49,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f 2023-03-28 20:55:49,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f, entries=150, sequenceid=315, filesize=11.9 K 2023-03-28 20:55:49,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 688ef3a4c2e822fe7803c6aa582a3052 in 79ms, sequenceid=315, compaction requested=true 2023-03-28 20:55:49,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:49,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:49,378 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:49,378 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:49,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:49,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:49,380 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39426 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:49,380 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:49,380 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,380 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/4feb7fe34b1d48cd87548289446b1baa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=38.5 K 2023-03-28 20:55:49,380 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:49,380 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:49,380 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,381 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/0963c25c2b3647deaab1bc2c9868ee90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=36.1 K 2023-03-28 20:55:49,381 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4feb7fe34b1d48cd87548289446b1baa, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680036944989 2023-03-28 20:55:49,381 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0963c25c2b3647deaab1bc2c9868ee90, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680036944989 2023-03-28 20:55:49,381 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 526745725c30427080f4c660c35eb4d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680036947146 2023-03-28 20:55:49,381 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1ac2080198b746649bab464774444273, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680036947146 2023-03-28 20:55:49,381 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting cd034fad1d714b8491ca04cf78ce5d73, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949291 2023-03-28 20:55:49,382 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d7d9f9ccd77a46b98ad6778b5eca5d9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949294 2023-03-28 20:55:49,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-28 20:55:49,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:49,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:49,392 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:49,405 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:49,418 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/091d5e54da1143b2b54a12bf3e7894dc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/091d5e54da1143b2b54a12bf3e7894dc 2023-03-28 20:55:49,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/51b37d7e7ea14bf8be4d9a74210df1c5 2023-03-28 20:55:49,419 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:49,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:49,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:49,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:49,423 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:49,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:49,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:49,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,423 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037009410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,426 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/43a5feff6cf740839d3be844fa9bbc9e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/43a5feff6cf740839d3be844fa9bbc9e 2023-03-28 20:55:49,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,426 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:49,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:49,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,433 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 091d5e54da1143b2b54a12bf3e7894dc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:49,433 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:49,433 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036949378; duration=0sec 2023-03-28 20:55:49,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037009421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,433 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:49,433 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:49,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46560 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:49,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:49,435 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,435 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a5512eb4f15c4f06aa7c8c8b553be53e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=45.5 K 2023-03-28 20:55:49,435 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a5512eb4f15c4f06aa7c8c8b553be53e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680036944981 2023-03-28 20:55:49,436 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f657eeaeab924bcc8422eff1b1e16649, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680036947087 2023-03-28 20:55:49,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037009430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:49,437 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,437 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:49,440 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e0501ecdee4c464482e40cf77ca5bdd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680036947146 2023-03-28 20:55:49,440 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,441 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 9b9e2d66f6164dbd8b1c4fa0e2f4020f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949294 2023-03-28 20:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,442 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into 43a5feff6cf740839d3be844fa9bbc9e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:49,442 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:49,442 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036949378; duration=0sec 2023-03-28 20:55:49,442 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6b6466[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:49,448 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:49,448 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,451 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#499 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:49,456 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:49,456 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,456 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:49,458 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:49,462 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:49,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:49,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:49,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:49,462 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:49,464 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,464 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,464 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,465 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,466 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:49,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,472 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:49,471 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:49,472 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037009468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58734 deadline: 1680037009469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037009525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037009534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,572 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:49,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:49,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:49,575 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:49,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:49,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:49,578 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:49,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:49,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:49,580 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0d67fb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:49,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:49,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,584 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:49,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,585 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:49,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:49,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:49,589 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:49,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:49,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:49,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:49,589 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:49,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,590 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,591 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,595 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:49,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,596 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:49,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:49,596 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:49,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037009637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58744 deadline: 1680037009674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037009679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,696 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:49,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:49,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:49,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:49,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:49,699 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:49,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:49,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:49,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,701 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:49,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:49,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:49,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:49,703 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,703 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:49,703 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:49,703 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:49,706 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,706 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e79407d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:49,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:49,710 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:49,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:49,714 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:49,717 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:49,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:49,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:49,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:49,717 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:49,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,720 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,728 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:49,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:49,729 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:49,730 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037009731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037009743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,830 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:49,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:49,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:49,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:49,834 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:49,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:49,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:49,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,835 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:49,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:49,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:49,852 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,852 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:49,852 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:49,852 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:49,853 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:49,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,854 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:49,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ecaf90c8a2d64acf99acb6bd18304710 2023-03-28 20:55:49,868 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/ee254b7a4be44a059c4a0136472e8138 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/ee254b7a4be44a059c4a0136472e8138 2023-03-28 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c7e486[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:49,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:49,871 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,873 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:49,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:49,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:49,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,874 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:49,876 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into ee254b7a4be44a059c4a0136472e8138(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:49,876 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:49,877 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=12, startTime=1680036949379; duration=0sec 2023-03-28 20:55:49,877 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:49,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:49,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,879 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:49,880 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:49,883 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:49,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:49,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:49,883 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:49,883 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:49,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:49,886 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,887 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:49,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:49,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:49,890 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:49,890 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:49,890 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:49,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:49,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:49,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58772 deadline: 1680037009945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037009978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037009984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,990 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:49,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:49,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:49,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:49,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:49,994 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:49,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:49,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:49,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,996 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:49,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:49,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:49,997 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:49,997 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:49,998 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:49,998 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:49,998 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:50,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:50,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:50,002 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:50,003 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:50,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273efa6c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:50,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:50,005 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,006 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:50,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,006 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:50,008 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:50,011 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:50,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:50,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:50,011 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:50,011 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:50,014 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,015 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,015 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,018 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:50,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:50,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,019 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,019 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:50,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037010035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037010050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,119 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:50,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:50,122 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:50,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:50,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:50,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,124 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,125 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,125 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,125 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:50,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:50,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d63cf[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:50,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:50,130 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,133 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,133 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,134 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:50,137 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:50,137 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:50,138 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,138 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,138 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,139 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,140 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,144 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:50,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:50,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,144 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,144 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:50,144 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:50,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:50,245 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:50,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:50,248 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,251 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,251 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,251 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,251 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:50,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:50,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:50,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:50,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:50,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:50,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f3a1df[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:50,256 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:50,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,258 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:50,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,260 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:50,263 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:50,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:50,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:50,263 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:50,263 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:50,266 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,266 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,266 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,268 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:50,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,273 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,273 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/70dd221a784d4131b5dd83f34d978666 2023-03-28 20:55:50,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/51b37d7e7ea14bf8be4d9a74210df1c5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5 2023-03-28 20:55:50,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5, entries=200, sequenceid=351, filesize=14.3 K 2023-03-28 20:55:50,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ecaf90c8a2d64acf99acb6bd18304710 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710 2023-03-28 20:55:50,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710, entries=150, sequenceid=351, filesize=11.9 K 2023-03-28 20:55:50,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/70dd221a784d4131b5dd83f34d978666 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666 2023-03-28 20:55:50,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666, entries=150, sequenceid=351, filesize=11.9 K 2023-03-28 20:55:50,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 688ef3a4c2e822fe7803c6aa582a3052 in 924ms, sequenceid=351, compaction requested=false 2023-03-28 20:55:50,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:50,373 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:50,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:50,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:50,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:50,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:50,376 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:50,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:50,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:50,377 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,377 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,378 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,379 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:50,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:50,380 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:50,380 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:50,388 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/a1d13babea924e0080a56a9e1e8b7d1f 2023-03-28 20:55:50,400 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/31326a7cb540482386bf1bfc6172f0d4 2023-03-28 20:55:50,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:50,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036950488, max timestamp=1680036950502Num uniques 859; 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036950488, max timestamp=1680036950502Num uniques 859; 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036950488, max timestamp=1680036950502Num uniques 859; 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:50,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037010535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037010542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037010544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58746 deadline: 1680037010545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037010558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037010645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037010645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037010650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,811 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b92674bcb7894ddb94ca0b166897d7e7 2023-03-28 20:55:50,815 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/a1d13babea924e0080a56a9e1e8b7d1f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f 2023-03-28 20:55:50,819 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f, entries=100, sequenceid=363, filesize=9.5 K 2023-03-28 20:55:50,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/31326a7cb540482386bf1bfc6172f0d4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4 2023-03-28 20:55:50,824 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4, entries=100, sequenceid=363, filesize=9.5 K 2023-03-28 20:55:50,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b92674bcb7894ddb94ca0b166897d7e7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7 2023-03-28 20:55:50,828 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7, entries=100, sequenceid=363, filesize=9.5 K 2023-03-28 20:55:50,829 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 688ef3a4c2e822fe7803c6aa582a3052 in 449ms, sequenceid=363, compaction requested=true 2023-03-28 20:55:50,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:50,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:50,831 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3992d3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:50,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:50,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,834 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:50,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,834 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:50,837 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:50,839 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:50,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:50,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:50,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:50,840 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:50,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,845 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,846 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:50,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:50,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,850 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,851 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:50,851 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:50,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036950533, max timestamp=1680036950856Num uniques 1068; 2023-03-28 20:55:50,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680036950533, max timestamp=1680036950856Num uniques 1068; 2023-03-28 20:55:50,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-28 20:55:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:50,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-28 20:55:50,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=67.09 KB heapSize=183.88 KB 2023-03-28 20:55:50,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:50,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:50,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:50,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:50,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:50,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:50,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037010891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037010892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037010899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,951 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:50,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:50,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:50,954 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:50,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:50,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:50,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,955 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,956 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:50,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:50,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,957 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:50,957 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,957 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:50,957 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:50,957 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:50,958 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:50,960 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,961 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b84e442[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:50,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:50,962 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,964 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:50,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:50,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:50,969 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:50,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:50,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:50,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:50,969 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:50,970 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,970 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,970 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:50,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,974 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:50,981 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:50,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:50,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:50,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:50,982 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:50,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:50,982 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:50,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:50,982 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037011000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037011000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037011007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,082 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,086 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,086 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,088 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,088 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,089 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,091 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:51,091 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:51,092 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@365ac7e6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:51,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:51,095 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:51,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:51,102 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:51,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:51,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:51,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:51,102 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:51,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,104 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,113 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,113 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:51,113 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:51,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037011206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037011206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,213 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,220 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,220 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037011216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,222 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:51,225 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:51,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:51,226 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@91875ab[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:51,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:51,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:51,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:51,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:51,234 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:51,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:51,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:51,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:51,234 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:51,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,237 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:51,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:51,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:51,242 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:51,243 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:51,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/96ce75a5eede4cf4a627ddbe471d928c 2023-03-28 20:55:51,343 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,346 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,348 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,351 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,351 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,351 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:51,351 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:51,352 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,353 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,353 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e70d151[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:51,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:51,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,358 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:51,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,358 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:51,360 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:51,362 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:51,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:51,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:51,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:51,362 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:51,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,364 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,364 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,366 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,369 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:51,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:51,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:51,368 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,369 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:51,369 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:51,369 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,469 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,473 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,473 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,475 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,475 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,476 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,477 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,477 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,477 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:51,478 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:51,480 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32aa9a29[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:51,481 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:51,482 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,483 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:51,484 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:51,484 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,484 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:51,485 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:51,488 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:51,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:51,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:51,488 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:51,488 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:51,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,491 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,492 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:51,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:51,496 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:51,496 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:51,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037011521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037011521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037011523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58746 deadline: 1680037011557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:51,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037011570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,596 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,600 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,600 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,601 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,601 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,604 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,604 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,605 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,605 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:51,606 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:51,608 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af1e6d[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:51,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:51,610 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:51,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,614 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:51,617 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:51,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:51,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:51,617 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:51,617 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:51,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,618 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:51,619 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,620 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,623 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,624 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:51,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:51,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:51,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:51,624 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,624 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:51,624 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:51,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/880736395515479eaff87c2e9e9e7816 2023-03-28 20:55:51,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/6fcc00dda91e49eb9010d20bc6b2bab0 2023-03-28 20:55:51,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/96ce75a5eede4cf4a627ddbe471d928c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c 2023-03-28 20:55:51,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c, entries=150, sequenceid=401, filesize=11.9 K 2023-03-28 20:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/880736395515479eaff87c2e9e9e7816 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816 2023-03-28 20:55:51,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816, entries=150, sequenceid=401, filesize=11.9 K 2023-03-28 20:55:51,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/6fcc00dda91e49eb9010d20bc6b2bab0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0 2023-03-28 20:55:51,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0, entries=150, sequenceid=401, filesize=11.9 K 2023-03-28 20:55:51,724 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:51,725 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~183.83 KB/188240, currentSize=114.05 KB/116790 for 688ef3a4c2e822fe7803c6aa582a3052 in 866ms, sequenceid=401, compaction requested=true 2023-03-28 20:55:51,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:51,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:51,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:51,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:51,725 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:51,725 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:51,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-28 20:55:51,728 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49320 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:51,728 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:51,728 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,728 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/091d5e54da1143b2b54a12bf3e7894dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=48.2 K 2023-03-28 20:55:51,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:51,729 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 091d5e54da1143b2b54a12bf3e7894dc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949294 2023-03-28 20:55:51,729 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 51b37d7e7ea14bf8be4d9a74210df1c5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680036949364 2023-03-28 20:55:51,730 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:51,730 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:51,730 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,730 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/43a5feff6cf740839d3be844fa9bbc9e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=45.8 K 2023-03-28 20:55:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:51,730 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:51,730 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a1d13babea924e0080a56a9e1e8b7d1f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680036949408 2023-03-28 20:55:51,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,730 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 43a5feff6cf740839d3be844fa9bbc9e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949294 2023-03-28 20:55:51,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:51,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:51,731 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 96ce75a5eede4cf4a627ddbe471d928c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:51,731 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ecaf90c8a2d64acf99acb6bd18304710, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680036949364 2023-03-28 20:55:51,731 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 31326a7cb540482386bf1bfc6172f0d4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680036949408 2023-03-28 20:55:51,732 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 880736395515479eaff87c2e9e9e7816, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:51,742 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,742 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,744 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:51,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:51,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:51,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:51,745 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:51,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:51,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:51,761 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:51,776 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,776 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:51,776 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:55:51,776 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:51,776 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:51,776 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:51,777 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:51,777 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:51,777 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:51,802 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/3c4cbadfad594460b18d082f5aec4e86 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3c4cbadfad594460b18d082f5aec4e86 2023-03-28 20:55:51,811 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into 3c4cbadfad594460b18d082f5aec4e86(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:51,811 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:51,811 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=12, startTime=1680036951725; duration=0sec 2023-03-28 20:55:51,811 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:51,812 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:51,813 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:51,813 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:51,813 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:51,813 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/ee254b7a4be44a059c4a0136472e8138, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=45.7 K 2023-03-28 20:55:51,813 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting ee254b7a4be44a059c4a0136472e8138, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680036949294 2023-03-28 20:55:51,814 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 70dd221a784d4131b5dd83f34d978666, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680036949364 2023-03-28 20:55:51,814 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b92674bcb7894ddb94ca0b166897d7e7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680036949408 2023-03-28 20:55:51,814 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 6fcc00dda91e49eb9010d20bc6b2bab0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:51,817 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/47cdb41bc2344143ba1945930d3f0028 2023-03-28 20:55:51,830 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#510 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:52,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:52,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:52,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037012048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037012049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037012054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037012159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037012162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037012162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,179 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/c7cd06d955b64078857be54a594ec213 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c7cd06d955b64078857be54a594ec213 2023-03-28 20:55:52,187 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into c7cd06d955b64078857be54a594ec213(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:52,187 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:52,187 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=12, startTime=1680036951725; duration=0sec 2023-03-28 20:55:52,187 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:52,261 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/d1a926c8e18540bea03a8aa1d132a775 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d1a926c8e18540bea03a8aa1d132a775 2023-03-28 20:55:52,265 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into d1a926c8e18540bea03a8aa1d132a775(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:52,266 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:52,266 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=12, startTime=1680036951725; duration=0sec 2023-03-28 20:55:52,266 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:52,268 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ce0cd6c169b443d49551a271932568c8 2023-03-28 20:55:52,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037012369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037012370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037012371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037012673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037012676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:52,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037012677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,704 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/fbd9edef5eab41b3a4dfeff1e7289b56 2023-03-28 20:55:52,708 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/47cdb41bc2344143ba1945930d3f0028 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028 2023-03-28 20:55:52,711 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028, entries=150, sequenceid=421, filesize=11.9 K 2023-03-28 20:55:52,712 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ce0cd6c169b443d49551a271932568c8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8 2023-03-28 20:55:52,714 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8, entries=150, sequenceid=421, filesize=11.9 K 2023-03-28 20:55:52,715 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/fbd9edef5eab41b3a4dfeff1e7289b56 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56 2023-03-28 20:55:52,718 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56, entries=150, sequenceid=421, filesize=11.9 K 2023-03-28 20:55:52,718 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 688ef3a4c2e822fe7803c6aa582a3052 in 942ms, sequenceid=421, compaction requested=false 2023-03-28 20:55:52,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:52,718 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:52,720 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,721 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d09cbe[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:52,722 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:52,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:52,725 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:52,728 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:52,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:52,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:52,728 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:52,728 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:52,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:52,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,729 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:52,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:52,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:52,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:52,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:52,733 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:52,734 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,734 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:52,734 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:52,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:52,834 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:52,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:52,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:52,837 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:52,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:52,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:52,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:52,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,837 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:52,838 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:52,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:52,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:52,839 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:52,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:52,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:52,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:52,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:52,840 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:53,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:53,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037013215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037013216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037013217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,247 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/87fd64b9cbd346639827c261d559e48d 2023-03-28 20:55:53,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037013322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037013322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037013323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037013527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037013527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037013527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58746 deadline: 1680037013580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,585 DEBUG [Thread-3342] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:53,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58812 deadline: 1680037013597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,602 DEBUG [Thread-3348] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., hostname=jenkins-hbase4.apache.org,42693,1680036815956, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-28 20:55:53,660 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cc52368272a04c5b8d82a96954b88720 2023-03-28 20:55:53,677 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f9604d3071d0417f8966cfa6adc0fff3 2023-03-28 20:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/87fd64b9cbd346639827c261d559e48d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d 2023-03-28 20:55:53,691 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:55:53,692 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cc52368272a04c5b8d82a96954b88720 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720 2023-03-28 20:55:53,700 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:55:53,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/f9604d3071d0417f8966cfa6adc0fff3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3 2023-03-28 20:55:53,704 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3, entries=150, sequenceid=436, filesize=11.9 K 2023-03-28 20:55:53,705 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 688ef3a4c2e822fe7803c6aa582a3052 in 865ms, sequenceid=436, compaction requested=true 2023-03-28 20:55:53,705 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:53,705 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:53,707 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50623cac[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:53,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:53,709 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,711 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:53,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,712 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:53,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:53,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:53,716 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:53,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:53,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:53,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:53,716 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:53,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,719 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,720 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:53,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:53,723 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:53,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,724 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:53,724 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:53,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:53,824 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:53,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:53,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:53,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:53,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:53,827 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:53,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:53,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:53,827 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:53,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,828 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,830 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:53,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:53,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,830 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:53,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:53,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-28 20:55:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:53,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:53,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:53,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:53,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:53,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:53,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ff9f5b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:53,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:53,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,848 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:53,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:53,850 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:53,854 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:53,854 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:53,855 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,855 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/dbcc5dd7087148eea94a09a412b0b52b 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,863 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:53,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:53,864 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:53,864 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:53,863 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037013858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037013859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037013860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/1cf87a2962844bb4afae062eed8433b7 2023-03-28 20:55:53,964 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:53,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:53,967 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:53,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:53,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:53,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:53,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,968 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,969 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:53,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:53,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:53,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:53,971 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:53,971 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:53,971 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:53,971 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:53,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:53,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037013968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037013968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037013969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,974 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:53,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c09a39f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:53,977 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:53,977 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,978 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:53,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:53,979 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:53,982 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:53,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:53,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:53,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:53,982 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:53,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:53,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,983 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:53,984 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:53,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:53,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:53,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:53,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:53,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:53,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:53,989 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:53,989 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:53,989 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,089 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:54,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:54,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:54,092 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:54,092 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,092 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:54,092 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:54,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:54,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:54,095 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:54,097 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0f9cc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:54,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:54,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:54,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:54,102 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:54,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:54,105 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:54,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:54,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:54,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:54,105 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:54,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,106 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,107 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,108 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:54,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:54,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:54,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,111 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,111 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037014174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037014175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037014175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,211 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:54,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:54,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:54,214 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:54,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:54,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:54,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,214 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,216 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:54,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:54,221 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41663228[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:54,222 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:54,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,224 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:54,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,228 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:54,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:54,230 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:54,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:54,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:54,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:54,231 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:54,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,236 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,237 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:54,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:54,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:54,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,238 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:54,238 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/be710f0d217e4785bda6080fc9ccc95e 2023-03-28 20:55:54,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/dbcc5dd7087148eea94a09a412b0b52b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b 2023-03-28 20:55:54,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b, entries=200, sequenceid=457, filesize=14.3 K 2023-03-28 20:55:54,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/1cf87a2962844bb4afae062eed8433b7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7 2023-03-28 20:55:54,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7, entries=150, sequenceid=457, filesize=11.9 K 2023-03-28 20:55:54,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/be710f0d217e4785bda6080fc9ccc95e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e 2023-03-28 20:55:54,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e, entries=150, sequenceid=457, filesize=11.9 K 2023-03-28 20:55:54,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 688ef3a4c2e822fe7803c6aa582a3052 in 478ms, sequenceid=457, compaction requested=true 2023-03-28 20:55:54,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:54,314 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:54,314 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:54,315 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:54,315 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:54,315 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,316 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c7cd06d955b64078857be54a594ec213, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=50.7 K 2023-03-28 20:55:54,316 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:54,316 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:54,316 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,316 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3c4cbadfad594460b18d082f5aec4e86, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=48.3 K 2023-03-28 20:55:54,316 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c7cd06d955b64078857be54a594ec213, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3c4cbadfad594460b18d082f5aec4e86, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 47cdb41bc2344143ba1945930d3f0028, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680036950881 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ce0cd6c169b443d49551a271932568c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680036950881 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 87fd64b9cbd346639827c261d559e48d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036952044 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cc52368272a04c5b8d82a96954b88720, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036952044 2023-03-28 20:55:54,317 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting dbcc5dd7087148eea94a09a412b0b52b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:54,318 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1cf87a2962844bb4afae062eed8433b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:54,326 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#519 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:54,335 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#520 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:54,338 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:54,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:54,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:54,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:54,343 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:54,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:54,343 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:54,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,344 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,345 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,346 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:54,346 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:54,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:54,348 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,348 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:54,348 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:55:54,348 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:54,348 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:54,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:54,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:54,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:54,349 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:54,363 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/3e00bd4ab70a4deda43ddca53a052597 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e00bd4ab70a4deda43ddca53a052597 2023-03-28 20:55:54,369 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 3e00bd4ab70a4deda43ddca53a052597(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:54,369 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:54,369 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=12, startTime=1680036954314; duration=0sec 2023-03-28 20:55:54,369 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:54,369 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:55:54,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:55:54,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:54,371 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,371 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d1a926c8e18540bea03a8aa1d132a775, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=48.3 K 2023-03-28 20:55:54,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d1a926c8e18540bea03a8aa1d132a775, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680036950533 2023-03-28 20:55:54,373 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting fbd9edef5eab41b3a4dfeff1e7289b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680036950881 2023-03-28 20:55:54,373 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting f9604d3071d0417f8966cfa6adc0fff3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680036952044 2023-03-28 20:55:54,373 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting be710f0d217e4785bda6080fc9ccc95e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:54,380 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:54,384 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9 2023-03-28 20:55:54,407 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ad6c9e2eeb2b4a648a211d010bc5e703 2023-03-28 20:55:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:54,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:54,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036954485, max timestamp=1680036954523Num uniques 859; 2023-03-28 20:55:54,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:55:54,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037014532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037014532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037014539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037014640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037014640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037014646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,748 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/382f50ff3a4b448eaf386ec37e16e6c7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/382f50ff3a4b448eaf386ec37e16e6c7 2023-03-28 20:55:54,752 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into 382f50ff3a4b448eaf386ec37e16e6c7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:54,752 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:54,753 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=12, startTime=1680036954314; duration=0sec 2023-03-28 20:55:54,753 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:54,791 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/82adc7d1cb8542528144a0bbd9d87d32 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/82adc7d1cb8542528144a0bbd9d87d32 2023-03-28 20:55:54,795 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into 82adc7d1cb8542528144a0bbd9d87d32(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:54,795 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:54,795 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=12, startTime=1680036954314; duration=0sec 2023-03-28 20:55:54,796 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:54,820 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/1099b95938c44401b606e93fdc0e7925 2023-03-28 20:55:54,825 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9 2023-03-28 20:55:54,828 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9, entries=100, sequenceid=468, filesize=9.5 K 2023-03-28 20:55:54,829 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/ad6c9e2eeb2b4a648a211d010bc5e703 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703 2023-03-28 20:55:54,833 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703, entries=100, sequenceid=468, filesize=9.5 K 2023-03-28 20:55:54,833 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/1099b95938c44401b606e93fdc0e7925 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925 2023-03-28 20:55:54,836 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925, entries=100, sequenceid=468, filesize=9.5 K 2023-03-28 20:55:54,837 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=120.76 KB/123660 for 688ef3a4c2e822fe7803c6aa582a3052 in 489ms, sequenceid=468, compaction requested=false 2023-03-28 20:55:54,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:54,837 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:54,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b296818[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:54,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:54,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:54,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:54,845 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:54,849 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:54,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:54,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:54,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:54,849 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:54,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:54,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=127.47 KB heapSize=371.88 KB 2023-03-28 20:55:54,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:54,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:54,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:54,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,852 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,857 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,860 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:54,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,860 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:54,860 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,860 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037014866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037014872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037014873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,960 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:54,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:54,964 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,966 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,966 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,967 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:54,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:54,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:54,968 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,968 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:54,969 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:54,969 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:54,971 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6d37b9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:54,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:54,973 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,975 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:54,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:54,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037014974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037014976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:54,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:54,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:54,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:54,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:54,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:54,984 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,984 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,984 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:54,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,987 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037014979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:54,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:54,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:54,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:54,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:54,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:54,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:54,994 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:54,994 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:55,094 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:55,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:55,100 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:55,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:55,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:55,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,102 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,103 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:55,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:55,108 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,108 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:55,108 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:55,108 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:55,110 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d84dba1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:55,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:55,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,118 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:55,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:55,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:55,120 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:55,122 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:55,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:55,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:55,123 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:55,123 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:55,125 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,125 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,125 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,127 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,129 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,136 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:55,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:55,136 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,137 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:55,137 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:55,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037015180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037015181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037015190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,237 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:55,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:55,240 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:55,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:55,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:55,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:55,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:55,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:55,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:55,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:55,244 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:55,244 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:55,246 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:55,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d83554b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:55,247 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:55,247 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,248 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:55,249 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,249 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:55,251 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:55,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:55,255 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:55,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:55,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:55,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:55,255 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:55,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,257 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,258 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,260 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,260 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,261 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,261 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:55,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:55,262 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:55,262 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:55,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/a41dcc6ce8e2492da4a6c83de0429633 2023-03-28 20:55:55,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/3936cb6ff9644d888f0655f6b96292b1 2023-03-28 20:55:55,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/761e4f8107b44ef096c5010ac234d812 2023-03-28 20:55:55,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/a41dcc6ce8e2492da4a6c83de0429633 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633 2023-03-28 20:55:55,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633, entries=150, sequenceid=497, filesize=11.9 K 2023-03-28 20:55:55,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/3936cb6ff9644d888f0655f6b96292b1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1 2023-03-28 20:55:55,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1, entries=150, sequenceid=497, filesize=11.9 K 2023-03-28 20:55:55,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/761e4f8107b44ef096c5010ac234d812 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812 2023-03-28 20:55:55,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812, entries=150, sequenceid=497, filesize=11.9 K 2023-03-28 20:55:55,310 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~371.83 KB/380752, currentSize=46.96 KB/48090 for 688ef3a4c2e822fe7803c6aa582a3052 in 459ms, sequenceid=497, compaction requested=true 2023-03-28 20:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:55,310 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:55,310 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:55,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:55,311 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:55,311 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:55,311 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,311 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:55,311 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e00bd4ab70a4deda43ddca53a052597, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=34.1 K 2023-03-28 20:55:55,312 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:55,312 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,312 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/382f50ff3a4b448eaf386ec37e16e6c7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=34.1 K 2023-03-28 20:55:55,312 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 3e00bd4ab70a4deda43ddca53a052597, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:55,312 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 382f50ff3a4b448eaf386ec37e16e6c7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:55,313 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1b1cd3ac5c35437aad8f5e0a5da8c3e9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680036953856 2023-03-28 20:55:55,313 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ad6c9e2eeb2b4a648a211d010bc5e703, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680036953856 2023-03-28 20:55:55,319 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3936cb6ff9644d888f0655f6b96292b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:55,319 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a41dcc6ce8e2492da4a6c83de0429633, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:55,329 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:55,331 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:55,343 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/fc556e1f9f964bb6828edcfdea9962d0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/fc556e1f9f964bb6828edcfdea9962d0 2023-03-28 20:55:55,347 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into fc556e1f9f964bb6828edcfdea9962d0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:55,347 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:55,347 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036955310; duration=0sec 2023-03-28 20:55:55,347 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:55,347 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:55,348 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34932 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:55,348 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:55,348 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,348 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/82adc7d1cb8542528144a0bbd9d87d32, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=34.1 K 2023-03-28 20:55:55,349 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 82adc7d1cb8542528144a0bbd9d87d32, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680036953214 2023-03-28 20:55:55,349 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1099b95938c44401b606e93fdc0e7925, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680036953856 2023-03-28 20:55:55,350 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 761e4f8107b44ef096c5010ac234d812, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:55,355 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:55,362 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:55,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:55,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:55,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:55,365 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,365 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,367 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,367 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,367 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:55,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:55,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:55,372 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:55,373 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:55,373 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:55,373 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:55,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:55,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036955511, max timestamp=1680036955541Num uniques 859; 2023-03-28 20:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036955511, max timestamp=1680036955541Num uniques 859; 2023-03-28 20:55:55,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036955511, max timestamp=1680036955541Num uniques 859; 2023-03-28 20:55:55,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:55,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:55,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:55:55,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037015578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037015582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037015586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037015688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037015688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037015694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,754 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/39c46340bf704753ade03d38937ff48e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/39c46340bf704753ade03d38937ff48e 2023-03-28 20:55:55,763 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 39c46340bf704753ade03d38937ff48e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:55,763 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:55,763 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036955310; duration=0sec 2023-03-28 20:55:55,763 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:55,772 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/d04c265a0e0243b6ba68a6bc03a8028d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d04c265a0e0243b6ba68a6bc03a8028d 2023-03-28 20:55:55,777 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into d04c265a0e0243b6ba68a6bc03a8028d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:55,777 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:55,777 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=13, startTime=1680036955311; duration=0sec 2023-03-28 20:55:55,777 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:55,793 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/d0181b67c3cc402b99e29dfb6ae32819 2023-03-28 20:55:55,832 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cbcd0b8e51b0493a808899b38ab3c932 2023-03-28 20:55:55,874 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b5606b83d57744679d9f8651e1e848c4 2023-03-28 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/d0181b67c3cc402b99e29dfb6ae32819 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819 2023-03-28 20:55:55,884 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819, entries=150, sequenceid=508, filesize=11.9 K 2023-03-28 20:55:55,885 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/cbcd0b8e51b0493a808899b38ab3c932 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932 2023-03-28 20:55:55,898 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932, entries=150, sequenceid=508, filesize=11.9 K 2023-03-28 20:55:55,900 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b5606b83d57744679d9f8651e1e848c4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4 2023-03-28 20:55:55,906 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4, entries=150, sequenceid=508, filesize=11.9 K 2023-03-28 20:55:55,906 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 688ef3a4c2e822fe7803c6aa582a3052 in 534ms, sequenceid=508, compaction requested=false 2023-03-28 20:55:55,907 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:55,907 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,910 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2a814b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:55,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:55,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,915 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:55,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,916 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:55,916 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:55,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-28 20:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:55,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:55,924 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:55,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:55,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:55,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:55,924 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:55,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,927 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:55,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:55,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:55,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:55,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:55,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:55,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:55,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:55,933 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:55,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:55,933 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:55,933 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:55,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037015944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037015944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037015947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:55,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/0c81134153344ab9bde3a2da7db7b2cf 2023-03-28 20:55:56,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bbd74478ff284d9a992179085a261ecd 2023-03-28 20:55:56,033 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,043 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,043 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/fcda435de89e41f2a8b26428921624c2 2023-03-28 20:55:56,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,045 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/0c81134153344ab9bde3a2da7db7b2cf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf 2023-03-28 20:55:56,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,057 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf, entries=200, sequenceid=548, filesize=14.3 K 2023-03-28 20:55:56,057 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,057 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:56,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037016052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037016052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,059 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/bbd74478ff284d9a992179085a261ecd as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd 2023-03-28 20:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e49a3a8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:56,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:56,062 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:56,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:56,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd, entries=150, sequenceid=548, filesize=11.9 K 2023-03-28 20:55:56,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/fcda435de89e41f2a8b26428921624c2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2 2023-03-28 20:55:56,067 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:56,069 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:56,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:56,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:56,069 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:56,069 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:56,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,071 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2, entries=150, sequenceid=548, filesize=11.9 K 2023-03-28 20:55:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 688ef3a4c2e822fe7803c6aa582a3052 in 151ms, sequenceid=548, compaction requested=true 2023-03-28 20:55:56,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:56,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:56,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:56,075 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:56,075 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:56,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:56,076 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:56,076 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/A is initiating minor compaction (all files) 2023-03-28 20:55:56,076 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/A in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,076 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/39c46340bf704753ade03d38937ff48e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=39.0 K 2023-03-28 20:55:56,077 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:56,077 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/B is initiating minor compaction (all files) 2023-03-28 20:55:56,077 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/B in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,077 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/fc556e1f9f964bb6828edcfdea9962d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=36.6 K 2023-03-28 20:55:56,077 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 39c46340bf704753ade03d38937ff48e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,078 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,079 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,079 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:56,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:56,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:56,079 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:56,079 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,080 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fc556e1f9f964bb6828edcfdea9962d0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:56,080 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d0181b67c3cc402b99e29dfb6ae32819, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680036954852 2023-03-28 20:55:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,081 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cbcd0b8e51b0493a808899b38ab3c932, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680036954852 2023-03-28 20:55:56,081 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 0c81134153344ab9bde3a2da7db7b2cf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680036955568 2023-03-28 20:55:56,081 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bbd74478ff284d9a992179085a261ecd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680036955568 2023-03-28 20:55:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:56,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:55:56,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:56,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:56,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:56,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:56,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:56,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:56,097 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#B#compaction#537 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:56,101 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#A#compaction#539 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:56,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/87e4a976202b4e0b9f0155ed16470e0d 2023-03-28 20:55:56,150 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/2107c135ab084af285786ea311b80a0b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/2107c135ab084af285786ea311b80a0b 2023-03-28 20:55:56,159 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/B of 688ef3a4c2e822fe7803c6aa582a3052 into 2107c135ab084af285786ea311b80a0b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:56,159 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:56,159 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/B, priority=13, startTime=1680036956075; duration=0sec 2023-03-28 20:55:56,159 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:56,159 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:55:56,164 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:55:56,164 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 688ef3a4c2e822fe7803c6aa582a3052/C is initiating minor compaction (all files) 2023-03-28 20:55:56,165 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 688ef3a4c2e822fe7803c6aa582a3052/C in TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,165 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d04c265a0e0243b6ba68a6bc03a8028d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp, totalSize=36.6 K 2023-03-28 20:55:56,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d04c265a0e0243b6ba68a6bc03a8028d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680036954531 2023-03-28 20:55:56,165 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b5606b83d57744679d9f8651e1e848c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680036954852 2023-03-28 20:55:56,166 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fcda435de89e41f2a8b26428921624c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680036955568 2023-03-28 20:55:56,179 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,183 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 688ef3a4c2e822fe7803c6aa582a3052#C#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:55:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,184 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,187 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,188 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/5230e63e795947eb8a40cd98fdb8daf4 2023-03-28 20:55:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,198 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,198 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,198 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,198 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:56,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:56,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,220 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256b3213[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:56,223 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:56,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,225 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:56,225 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,225 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:56,227 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:56,230 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:56,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:56,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:56,230 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:56,230 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:56,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036956203, max timestamp=1680036956228Num uniques 859; 2023-03-28 20:55:56,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:56,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,233 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,238 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:56,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:56,239 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:56,239 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:56,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037016263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037016268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037016269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,339 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,348 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,348 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,349 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,349 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,350 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,350 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,351 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:56,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f502f8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:56,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:56,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,370 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:56,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,371 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:56,374 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:56,376 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:56,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:56,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:56,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:56,376 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:56,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037016374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,395 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,396 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,396 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,396 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,398 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037016392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037016392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,400 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,400 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,400 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,400 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,400 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,401 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,400 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,401 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:56,401 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:56,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:56,501 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,512 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,512 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,512 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,512 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,514 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,514 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,516 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,516 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:56,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,520 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,520 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,520 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,520 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:56,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2becf934[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:56,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:56,532 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,534 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,534 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,534 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:56,537 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,538 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/240156914d60412e9c380d4c1dc22ee3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/240156914d60412e9c380d4c1dc22ee3 2023-03-28 20:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:56,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:56,540 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:56,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:56,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:56,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:56,540 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:56,542 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/A of 688ef3a4c2e822fe7803c6aa582a3052 into 240156914d60412e9c380d4c1dc22ee3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:56,542 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:56,542 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/A, priority=13, startTime=1680036956075; duration=0sec 2023-03-28 20:55:56,543 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:56,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,544 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,551 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:56,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:56,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,551 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,552 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:56,552 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:56,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037016600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037016600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037016600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,622 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/2c543549329d4e0fafba88cf234771b7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/2c543549329d4e0fafba88cf234771b7 2023-03-28 20:55:56,626 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 688ef3a4c2e822fe7803c6aa582a3052/C of 688ef3a4c2e822fe7803c6aa582a3052 into 2c543549329d4e0fafba88cf234771b7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:55:56,626 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:56,626 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052., storeName=688ef3a4c2e822fe7803c6aa582a3052/C, priority=13, startTime=1680036956075; duration=0sec 2023-03-28 20:55:56,626 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:56,652 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/35ad012965104ae9a0574119c4ed8188 2023-03-28 20:55:56,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,659 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,660 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/87e4a976202b4e0b9f0155ed16470e0d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87e4a976202b4e0b9f0155ed16470e0d 2023-03-28 20:55:56,663 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,668 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,669 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87e4a976202b4e0b9f0155ed16470e0d, entries=200, sequenceid=558, filesize=14.3 K 2023-03-28 20:55:56,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/5230e63e795947eb8a40cd98fdb8daf4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/5230e63e795947eb8a40cd98fdb8daf4 2023-03-28 20:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,687 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382b4b02[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:56,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:56,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:56,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,690 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:56,692 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:56,694 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:56,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:56,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:56,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:56,695 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:56,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,696 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:56,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:56,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:56,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/5230e63e795947eb8a40cd98fdb8daf4, entries=150, sequenceid=558, filesize=11.9 K 2023-03-28 20:55:56,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/35ad012965104ae9a0574119c4ed8188 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/35ad012965104ae9a0574119c4ed8188 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:56,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:56,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:56,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:56,702 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:56,703 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:56,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:56,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/35ad012965104ae9a0574119c4ed8188, entries=150, sequenceid=558, filesize=11.9 K 2023-03-28 20:55:56,712 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-28 20:55:56,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=129.71 KB/132820 for 688ef3a4c2e822fe7803c6aa582a3052 in 625ms, sequenceid=558, compaction requested=false 2023-03-28 20:55:56,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:56,803 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:56,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:56,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:56,807 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:56,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,807 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:56,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:56,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,810 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:56,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:56,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:56,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:56,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:56,819 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=129.71 KB heapSize=381.64 KB 2023-03-28 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:56,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:56,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:56,820 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:55:56,851 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/3e16d111bd0b4359bcf1fedec610f064 2023-03-28 20:55:56,898 WARN [DataStreamer for file /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7cd50ff34c8a41ceb956f561003785f2] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-28 20:55:56,898 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7cd50ff34c8a41ceb956f561003785f2 2023-03-28 20:55:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:56,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:56,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037016933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037016944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:56,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:56,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037016945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037017045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037017048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037017057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037017255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037017256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037017264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,329 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b1dcee83e1f24758b097b2efb8ea691c 2023-03-28 20:55:57,335 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/3e16d111bd0b4359bcf1fedec610f064 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e16d111bd0b4359bcf1fedec610f064 2023-03-28 20:55:57,339 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e16d111bd0b4359bcf1fedec610f064, entries=150, sequenceid=588, filesize=11.9 K 2023-03-28 20:55:57,340 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7cd50ff34c8a41ceb956f561003785f2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7cd50ff34c8a41ceb956f561003785f2 2023-03-28 20:55:57,345 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7cd50ff34c8a41ceb956f561003785f2, entries=150, sequenceid=588, filesize=11.9 K 2023-03-28 20:55:57,346 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/b1dcee83e1f24758b097b2efb8ea691c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b1dcee83e1f24758b097b2efb8ea691c 2023-03-28 20:55:57,365 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b1dcee83e1f24758b097b2efb8ea691c, entries=150, sequenceid=588, filesize=11.9 K 2023-03-28 20:55:57,366 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~381.59 KB/390752, currentSize=53.67 KB/54960 for 688ef3a4c2e822fe7803c6aa582a3052 in 547ms, sequenceid=588, compaction requested=true 2023-03-28 20:55:57,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:57,366 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:57,372 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:57,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:57,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c0dbe[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:57,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:57,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,375 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:57,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,376 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:57,378 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:57,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:57,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:57,378 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:57,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:57,380 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:57,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:57,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:57,380 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:57,380 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:57,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:57,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:57,383 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:57,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:57,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,390 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:57,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:57,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:57,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:57,391 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:57,391 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:57,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:55:57,491 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:57,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:57,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:55:57,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:55:57,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:55:57,495 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:55:57,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:57,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:55:57,495 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:55:57,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:57,497 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:57,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:57,497 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:57,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:55:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:57,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:55:57,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:55:57,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:55:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:55:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:55:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:55:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:55:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:55:57,508 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:57,508 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. started... 2023-03-28 20:55:57,508 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:57,547 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/7abdb530790542a986a902ee1eefb1e2 2023-03-28 20:55:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:57,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. as already flushing 2023-03-28 20:55:57,586 WARN [DataStreamer for file /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/8c8ba20ac3ba430495e8cf40dd2eb025] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-28 20:55:57,586 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/8c8ba20ac3ba430495e8cf40dd2eb025 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036957614, max timestamp=1680036957652Num uniques 859; 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036957614, max timestamp=1680036957652Num uniques 859; 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036957614, max timestamp=1680036957651Num uniques 859; 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:57,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:55:57,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58812 deadline: 1680037017684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037017688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037017690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037017691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58746 deadline: 1680037017698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58812 deadline: 1680037017800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037017800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037017806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037017806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:57,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58746 deadline: 1680037017813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:57,954 DEBUG [Thread-3351] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c050db8 to 127.0.0.1:56827 2023-03-28 20:55:57,954 DEBUG [Thread-3351] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:57,961 DEBUG [Thread-3359] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bbac0b5 to 127.0.0.1:56827 2023-03-28 20:55:57,961 DEBUG [Thread-3359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:57,964 DEBUG [Thread-3353] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28e3107b to 127.0.0.1:56827 2023-03-28 20:55:57,964 DEBUG [Thread-3353] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:57,964 DEBUG [Thread-3357] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21359a0e to 127.0.0.1:56827 2023-03-28 20:55:57,964 DEBUG [Thread-3357] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:57,973 DEBUG [Thread-3355] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14f9adf3 to 127.0.0.1:56827 2023-03-28 20:55:57,973 DEBUG [Thread-3355] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:58,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58734 deadline: 1680037018013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:58,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58812 deadline: 1680037018013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:58,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58744 deadline: 1680037018014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:58,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58772 deadline: 1680037018015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:55:58,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58746 deadline: 1680037018025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,051 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/d27e707636bd46bc9f000a06968eba5f 2023-03-28 20:55:58,055 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/7abdb530790542a986a902ee1eefb1e2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/7abdb530790542a986a902ee1eefb1e2 2023-03-28 20:55:58,062 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/7abdb530790542a986a902ee1eefb1e2, entries=150, sequenceid=599, filesize=11.9 K 2023-03-28 20:55:58,062 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/8c8ba20ac3ba430495e8cf40dd2eb025 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/8c8ba20ac3ba430495e8cf40dd2eb025 2023-03-28 20:55:58,067 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/8c8ba20ac3ba430495e8cf40dd2eb025, entries=150, sequenceid=599, filesize=11.9 K 2023-03-28 20:55:58,068 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/d27e707636bd46bc9f000a06968eba5f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d27e707636bd46bc9f000a06968eba5f 2023-03-28 20:55:58,071 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d27e707636bd46bc9f000a06968eba5f, entries=150, sequenceid=599, filesize=11.9 K 2023-03-28 20:55:58,072 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 688ef3a4c2e822fe7803c6aa582a3052 in 564ms, sequenceid=599, compaction requested=true 2023-03-28 20:55:58,072 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:58,072 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:55:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,265 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:58,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,266 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:58,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:55:58,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e742501[Count = 0] remaining members to acquire global barrier 2023-03-28 20:55:58,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:55:58,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:55:58,269 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,269 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:55:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:55:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:55:58,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:55:58,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:58,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:55:58,272 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:55:58,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:55:58,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:55:58,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:55:58,272 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:55:58,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:55:58,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:55:58,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:58,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:55:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:55:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:55:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:55:58,278 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:55:58,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:55:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:55:58,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:55:58,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:55:58,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:55:58,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:58,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:55:58,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:55:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:55:58,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,279 DEBUG [Thread-3350] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:55:58,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:55:58,279 DEBUG [Thread-3350] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:55:58,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:55:58,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:58,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:58,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:55:58,318 DEBUG [Thread-3348] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0db388d3 to 127.0.0.1:56827 2023-03-28 20:55:58,318 DEBUG [Thread-3340] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1df28b68 to 127.0.0.1:56827 2023-03-28 20:55:58,318 DEBUG [Thread-3340] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,318 DEBUG [Thread-3346] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14c209bc to 127.0.0.1:56827 2023-03-28 20:55:58,318 DEBUG [Thread-3346] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,318 DEBUG [Thread-3348] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,318 DEBUG [Thread-3344] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f0735dc to 127.0.0.1:56827 2023-03-28 20:55:58,319 DEBUG [Thread-3344] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/bac5f8598ced43909922f492d10989f7 2023-03-28 20:55:58,328 DEBUG [Thread-3342] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5cd71f to 127.0.0.1:56827 2023-03-28 20:55:58,328 DEBUG [Thread-3342] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,379 DEBUG [Thread-3350] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:55:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 2175 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 6524 rows 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 2184 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 6549 rows 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 2176 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 6528 rows 2023-03-28 20:55:58,380 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 2161 2023-03-28 20:55:58,381 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 6483 rows 2023-03-28 20:55:58,381 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(402): scanned 2195 2023-03-28 20:55:58,381 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(403): verified 6585 rows 2023-03-28 20:55:58,381 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:55:58,381 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18674076 to 127.0.0.1:56827 2023-03-28 20:55:58,381 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:55:58,382 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:55:58,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:55:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:55:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:55:58,387 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036958387"}]},"ts":"1680036958387"} 2023-03-28 20:55:58,388 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:55:58,391 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:55:58,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, UNASSIGN}] 2023-03-28 20:55:58,393 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, UNASSIGN 2023-03-28 20:55:58,393 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=688ef3a4c2e822fe7803c6aa582a3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:55:58,394 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036958393"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036958393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036958393"}]},"ts":"1680036958393"} 2023-03-28 20:55:58,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:55:58,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:55:58,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:55:58,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 688ef3a4c2e822fe7803c6aa582a3052, disabling compactions & flushes 2023-03-28 20:55:58,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:58,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:55:58,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/20ccceade6ce428c803453c4acc6800c 2023-03-28 20:55:58,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:55:59,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a3366ac5473640418101291a29b62c46 2023-03-28 20:55:59,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/bac5f8598ced43909922f492d10989f7 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/bac5f8598ced43909922f492d10989f7 2023-03-28 20:55:59,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/bac5f8598ced43909922f492d10989f7, entries=150, sequenceid=638, filesize=11.9 K 2023-03-28 20:55:59,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/20ccceade6ce428c803453c4acc6800c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/20ccceade6ce428c803453c4acc6800c 2023-03-28 20:55:59,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/20ccceade6ce428c803453c4acc6800c, entries=150, sequenceid=638, filesize=11.9 K 2023-03-28 20:55:59,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/a3366ac5473640418101291a29b62c46 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a3366ac5473640418101291a29b62c46 2023-03-28 20:55:59,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a3366ac5473640418101291a29b62c46, entries=150, sequenceid=638, filesize=11.9 K 2023-03-28 20:55:59,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=13.42 KB/13740 for 688ef3a4c2e822fe7803c6aa582a3052 in 844ms, sequenceid=638, compaction requested=true 2023-03-28 20:55:59,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:55:59,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:59,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:55:59,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:59,161 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. because compaction request was cancelled 2023-03-28 20:55:59,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. after waiting 0 ms 2023-03-28 20:55:59,161 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. because compaction request was cancelled 2023-03-28 20:55:59,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:59,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:55:59,161 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. because compaction request was cancelled 2023-03-28 20:55:59,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 688ef3a4c2e822fe7803c6aa582a3052 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-28 20:55:59,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:55:59,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=A 2023-03-28 20:55:59,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:59,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=B 2023-03-28 20:55:59,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:59,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 688ef3a4c2e822fe7803c6aa582a3052, store=C 2023-03-28 20:55:59,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:55:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:55:59,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/31cf3be40b734b9c81d95d6433bf725c 2023-03-28 20:55:59,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7f80d95a7c0e4c9eafa7fc2825d57266 2023-03-28 20:55:59,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/c58f04765f6a481c801b099b34f6cf7d 2023-03-28 20:55:59,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/A/31cf3be40b734b9c81d95d6433bf725c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/31cf3be40b734b9c81d95d6433bf725c 2023-03-28 20:55:59,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/31cf3be40b734b9c81d95d6433bf725c, entries=50, sequenceid=643, filesize=7.1 K 2023-03-28 20:55:59,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/B/7f80d95a7c0e4c9eafa7fc2825d57266 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7f80d95a7c0e4c9eafa7fc2825d57266 2023-03-28 20:55:59,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7f80d95a7c0e4c9eafa7fc2825d57266, entries=50, sequenceid=643, filesize=7.1 K 2023-03-28 20:55:59,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/.tmp/C/c58f04765f6a481c801b099b34f6cf7d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c58f04765f6a481c801b099b34f6cf7d 2023-03-28 20:55:59,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c58f04765f6a481c801b099b34f6cf7d, entries=50, sequenceid=643, filesize=7.1 K 2023-03-28 20:55:59,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 688ef3a4c2e822fe7803c6aa582a3052 in 837ms, sequenceid=643, compaction requested=true 2023-03-28 20:56:00,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/63a54e3b809b49aa9e8f9c4ead8b7e04, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/2357018e6e204a0c8862b48199b78fc2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d37565d1f6342e3bde4c019728e18a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/4feb7fe34b1d48cd87548289446b1baa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/091d5e54da1143b2b54a12bf3e7894dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c7cd06d955b64078857be54a594ec213, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e00bd4ab70a4deda43ddca53a052597, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/39c46340bf704753ade03d38937ff48e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf] to archive 2023-03-28 20:56:00,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:00,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d87f8808ad1461ba791eb1a6807d187 2023-03-28 20:56:00,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dcbc18c3caf44d8fbcf794b013adbf2d 2023-03-28 20:56:00,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/40f3f2d0bdb144e69ea4d6a8758cfb0d 2023-03-28 20:56:00,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/63a54e3b809b49aa9e8f9c4ead8b7e04 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/63a54e3b809b49aa9e8f9c4ead8b7e04 2023-03-28 20:56:00,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d4a31391cb8f41edb966796f15c10ac3 2023-03-28 20:56:00,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/46efe59250d6471eb97d03b92acb41e0 2023-03-28 20:56:00,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/fefbe3d290854347b6876ffb306e009f 2023-03-28 20:56:00,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0a0e11973506431c85d68873edab5007 2023-03-28 20:56:00,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/2357018e6e204a0c8862b48199b78fc2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/2357018e6e204a0c8862b48199b78fc2 2023-03-28 20:56:00,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/19b081ea322749a3a9cdd6b26d3f99c3 2023-03-28 20:56:00,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/6a98e11ec6ef49818206bea2b78c66f3 2023-03-28 20:56:00,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d37565d1f6342e3bde4c019728e18a5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/5d37565d1f6342e3bde4c019728e18a5 2023-03-28 20:56:00,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c808a597082648bca43913daf81946c3 2023-03-28 20:56:00,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/4feb7fe34b1d48cd87548289446b1baa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/4feb7fe34b1d48cd87548289446b1baa 2023-03-28 20:56:00,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3da61ebc9a344eccab3ee58f61b832cc 2023-03-28 20:56:00,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/526745725c30427080f4c660c35eb4d6 2023-03-28 20:56:00,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/cd034fad1d714b8491ca04cf78ce5d73 2023-03-28 20:56:00,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/091d5e54da1143b2b54a12bf3e7894dc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/091d5e54da1143b2b54a12bf3e7894dc 2023-03-28 20:56:00,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/51b37d7e7ea14bf8be4d9a74210df1c5 2023-03-28 20:56:00,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a1d13babea924e0080a56a9e1e8b7d1f 2023-03-28 20:56:00,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c7cd06d955b64078857be54a594ec213 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/c7cd06d955b64078857be54a594ec213 2023-03-28 20:56:00,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/96ce75a5eede4cf4a627ddbe471d928c 2023-03-28 20:56:00,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/47cdb41bc2344143ba1945930d3f0028 2023-03-28 20:56:00,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87fd64b9cbd346639827c261d559e48d 2023-03-28 20:56:00,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/dbcc5dd7087148eea94a09a412b0b52b 2023-03-28 20:56:00,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e00bd4ab70a4deda43ddca53a052597 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e00bd4ab70a4deda43ddca53a052597 2023-03-28 20:56:00,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/1b1cd3ac5c35437aad8f5e0a5da8c3e9 2023-03-28 20:56:00,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/39c46340bf704753ade03d38937ff48e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/39c46340bf704753ade03d38937ff48e 2023-03-28 20:56:00,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/a41dcc6ce8e2492da4a6c83de0429633 2023-03-28 20:56:00,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/d0181b67c3cc402b99e29dfb6ae32819 2023-03-28 20:56:00,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/0c81134153344ab9bde3a2da7db7b2cf 2023-03-28 20:56:00,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ff9fefe3d1a745d381976d45d7611d3d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ee41338741944b9a8ff9caa60a28d3db, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/c09b488a97c943c4b565f48e4be842d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/0963c25c2b3647deaab1bc2c9868ee90, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/43a5feff6cf740839d3be844fa9bbc9e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3c4cbadfad594460b18d082f5aec4e86, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/382f50ff3a4b448eaf386ec37e16e6c7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/fc556e1f9f964bb6828edcfdea9962d0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd] to archive 2023-03-28 20:56:00,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:00,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ec002e2b421a4e468df41baeed37731c 2023-03-28 20:56:00,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f818b0e3f60b423cb8164c5f5bc83790 2023-03-28 20:56:00,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ff9fefe3d1a745d381976d45d7611d3d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ff9fefe3d1a745d381976d45d7611d3d 2023-03-28 20:56:00,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/b70e3ae7447f423b81330834f46d496b 2023-03-28 20:56:00,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7488c82f34f047889dc98a4ac02d7f7a 2023-03-28 20:56:00,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/f849b4836f90417d863287d5581623cf 2023-03-28 20:56:00,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf9fb63f2f02417bb99a96e63feae29e 2023-03-28 20:56:00,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ee41338741944b9a8ff9caa60a28d3db to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ee41338741944b9a8ff9caa60a28d3db 2023-03-28 20:56:00,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bf46d4d981e44b19a3008ab59dbe1e44 2023-03-28 20:56:00,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/23155006531a41da84fc07b25e744f4a 2023-03-28 20:56:00,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/c09b488a97c943c4b565f48e4be842d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/c09b488a97c943c4b565f48e4be842d0 2023-03-28 20:56:00,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecdeb5f818be43fbbfd525cb688ab8ee 2023-03-28 20:56:00,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cd82ed9ac8bb422794f0eedf469f1eca 2023-03-28 20:56:00,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/0963c25c2b3647deaab1bc2c9868ee90 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/0963c25c2b3647deaab1bc2c9868ee90 2023-03-28 20:56:00,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/97bd70a337ae47e9a04833dc3560eacc 2023-03-28 20:56:00,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1ac2080198b746649bab464774444273 2023-03-28 20:56:00,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/43a5feff6cf740839d3be844fa9bbc9e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/43a5feff6cf740839d3be844fa9bbc9e 2023-03-28 20:56:00,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/d7d9f9ccd77a46b98ad6778b5eca5d9c 2023-03-28 20:56:00,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ecaf90c8a2d64acf99acb6bd18304710 2023-03-28 20:56:00,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/31326a7cb540482386bf1bfc6172f0d4 2023-03-28 20:56:00,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3c4cbadfad594460b18d082f5aec4e86 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3c4cbadfad594460b18d082f5aec4e86 2023-03-28 20:56:00,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/880736395515479eaff87c2e9e9e7816 2023-03-28 20:56:00,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ce0cd6c169b443d49551a271932568c8 2023-03-28 20:56:00,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cc52368272a04c5b8d82a96954b88720 2023-03-28 20:56:00,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/382f50ff3a4b448eaf386ec37e16e6c7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/382f50ff3a4b448eaf386ec37e16e6c7 2023-03-28 20:56:00,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/1cf87a2962844bb4afae062eed8433b7 2023-03-28 20:56:00,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/ad6c9e2eeb2b4a648a211d010bc5e703 2023-03-28 20:56:00,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/fc556e1f9f964bb6828edcfdea9962d0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/fc556e1f9f964bb6828edcfdea9962d0 2023-03-28 20:56:00,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/3936cb6ff9644d888f0655f6b96292b1 2023-03-28 20:56:00,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/cbcd0b8e51b0493a808899b38ab3c932 2023-03-28 20:56:00,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/bbd74478ff284d9a992179085a261ecd 2023-03-28 20:56:00,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f686a50705c14dc8871bc93fc4d3f9a5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/32296f6771e64d60bce42bcd42bb6a84, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a5512eb4f15c4f06aa7c8c8b553be53e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/ee254b7a4be44a059c4a0136472e8138, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d1a926c8e18540bea03a8aa1d132a775, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/82adc7d1cb8542528144a0bbd9d87d32, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d04c265a0e0243b6ba68a6bc03a8028d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2] to archive 2023-03-28 20:56:00,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:00,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f6a6b2fb67c24288ae4b388ae93c10cc 2023-03-28 20:56:00,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a52ec68eecf54b68bb291a05fce75c00 2023-03-28 20:56:00,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f686a50705c14dc8871bc93fc4d3f9a5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f686a50705c14dc8871bc93fc4d3f9a5 2023-03-28 20:56:00,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0d2c3a698b564a42819d69fe80b9d5b7 2023-03-28 20:56:00,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6d04cd31b56c46aeafe433f148f7f929 2023-03-28 20:56:00,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/20c54ce005ea4d0b92e355716c80a22a 2023-03-28 20:56:00,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/0ecb9c8d8db5484a8f1e9e68a0824872 2023-03-28 20:56:00,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/bf6b45206c524ab1a8de309dbc0841b9 2023-03-28 20:56:00,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/32296f6771e64d60bce42bcd42bb6a84 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/32296f6771e64d60bce42bcd42bb6a84 2023-03-28 20:56:00,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a81cae4acd27429ca8c7e4f041641cb2 2023-03-28 20:56:00,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70f4b646271d48a08c0ebd3a85001035 2023-03-28 20:56:00,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a5512eb4f15c4f06aa7c8c8b553be53e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a5512eb4f15c4f06aa7c8c8b553be53e 2023-03-28 20:56:00,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c3b4b72e4fd7479b8bad81b24f893438 2023-03-28 20:56:00,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f657eeaeab924bcc8422eff1b1e16649 2023-03-28 20:56:00,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/e0501ecdee4c464482e40cf77ca5bdd0 2023-03-28 20:56:00,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/ee254b7a4be44a059c4a0136472e8138 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/ee254b7a4be44a059c4a0136472e8138 2023-03-28 20:56:00,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/9b9e2d66f6164dbd8b1c4fa0e2f4020f 2023-03-28 20:56:00,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/70dd221a784d4131b5dd83f34d978666 2023-03-28 20:56:00,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b92674bcb7894ddb94ca0b166897d7e7 2023-03-28 20:56:00,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d1a926c8e18540bea03a8aa1d132a775 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d1a926c8e18540bea03a8aa1d132a775 2023-03-28 20:56:00,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/6fcc00dda91e49eb9010d20bc6b2bab0 2023-03-28 20:56:00,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fbd9edef5eab41b3a4dfeff1e7289b56 2023-03-28 20:56:00,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/f9604d3071d0417f8966cfa6adc0fff3 2023-03-28 20:56:00,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/82adc7d1cb8542528144a0bbd9d87d32 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/82adc7d1cb8542528144a0bbd9d87d32 2023-03-28 20:56:00,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/be710f0d217e4785bda6080fc9ccc95e 2023-03-28 20:56:00,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/1099b95938c44401b606e93fdc0e7925 2023-03-28 20:56:00,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d04c265a0e0243b6ba68a6bc03a8028d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d04c265a0e0243b6ba68a6bc03a8028d 2023-03-28 20:56:00,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/761e4f8107b44ef096c5010ac234d812 2023-03-28 20:56:00,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b5606b83d57744679d9f8651e1e848c4 2023-03-28 20:56:00,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/fcda435de89e41f2a8b26428921624c2 2023-03-28 20:56:00,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/recovered.edits/646.seqid, newMaxSeqId=646, maxSeqId=1 2023-03-28 20:56:00,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052. 2023-03-28 20:56:00,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 688ef3a4c2e822fe7803c6aa582a3052: 2023-03-28 20:56:00,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:56:00,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=688ef3a4c2e822fe7803c6aa582a3052, regionState=CLOSED 2023-03-28 20:56:00,118 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036960117"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036960117"}]},"ts":"1680036960117"} 2023-03-28 20:56:00,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-28 20:56:00,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 688ef3a4c2e822fe7803c6aa582a3052, server=jenkins-hbase4.apache.org,42693,1680036815956 in 1.7250 sec 2023-03-28 20:56:00,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-28 20:56:00,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=688ef3a4c2e822fe7803c6aa582a3052, UNASSIGN in 1.7280 sec 2023-03-28 20:56:00,121 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036960121"}]},"ts":"1680036960121"} 2023-03-28 20:56:00,122 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:56:00,124 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:56:00,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7420 sec 2023-03-28 20:56:00,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-28 20:56:00,493 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-28 20:56:00,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:56:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,496 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,496 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-28 20:56:00,501 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:56:00,503 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/recovered.edits] 2023-03-28 20:56:00,505 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/240156914d60412e9c380d4c1dc22ee3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/240156914d60412e9c380d4c1dc22ee3 2023-03-28 20:56:00,506 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/31cf3be40b734b9c81d95d6433bf725c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/31cf3be40b734b9c81d95d6433bf725c 2023-03-28 20:56:00,507 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e16d111bd0b4359bcf1fedec610f064 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/3e16d111bd0b4359bcf1fedec610f064 2023-03-28 20:56:00,507 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/7abdb530790542a986a902ee1eefb1e2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/7abdb530790542a986a902ee1eefb1e2 2023-03-28 20:56:00,508 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87e4a976202b4e0b9f0155ed16470e0d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/87e4a976202b4e0b9f0155ed16470e0d 2023-03-28 20:56:00,509 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/bac5f8598ced43909922f492d10989f7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/A/bac5f8598ced43909922f492d10989f7 2023-03-28 20:56:00,511 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/20ccceade6ce428c803453c4acc6800c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/20ccceade6ce428c803453c4acc6800c 2023-03-28 20:56:00,512 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/2107c135ab084af285786ea311b80a0b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/2107c135ab084af285786ea311b80a0b 2023-03-28 20:56:00,513 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/5230e63e795947eb8a40cd98fdb8daf4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/5230e63e795947eb8a40cd98fdb8daf4 2023-03-28 20:56:00,514 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7cd50ff34c8a41ceb956f561003785f2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7cd50ff34c8a41ceb956f561003785f2 2023-03-28 20:56:00,515 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7f80d95a7c0e4c9eafa7fc2825d57266 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/7f80d95a7c0e4c9eafa7fc2825d57266 2023-03-28 20:56:00,516 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/8c8ba20ac3ba430495e8cf40dd2eb025 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/B/8c8ba20ac3ba430495e8cf40dd2eb025 2023-03-28 20:56:00,517 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/2c543549329d4e0fafba88cf234771b7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/2c543549329d4e0fafba88cf234771b7 2023-03-28 20:56:00,518 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/35ad012965104ae9a0574119c4ed8188 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/35ad012965104ae9a0574119c4ed8188 2023-03-28 20:56:00,519 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a3366ac5473640418101291a29b62c46 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/a3366ac5473640418101291a29b62c46 2023-03-28 20:56:00,520 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b1dcee83e1f24758b097b2efb8ea691c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/b1dcee83e1f24758b097b2efb8ea691c 2023-03-28 20:56:00,521 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c58f04765f6a481c801b099b34f6cf7d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/c58f04765f6a481c801b099b34f6cf7d 2023-03-28 20:56:00,522 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d27e707636bd46bc9f000a06968eba5f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/C/d27e707636bd46bc9f000a06968eba5f 2023-03-28 20:56:00,524 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/recovered.edits/646.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052/recovered.edits/646.seqid 2023-03-28 20:56:00,524 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/688ef3a4c2e822fe7803c6aa582a3052 2023-03-28 20:56:00,524 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:56:00,526 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,527 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:56:00,529 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:56:00,530 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,530 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:56:00,530 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036960530"}]},"ts":"9223372036854775807"} 2023-03-28 20:56:00,531 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:56:00,531 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 688ef3a4c2e822fe7803c6aa582a3052, NAME => 'TestAcidGuarantees,,1680036936766.688ef3a4c2e822fe7803c6aa582a3052.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:56:00,531 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:56:00,531 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036960531"}]},"ts":"9223372036854775807"} 2023-03-28 20:56:00,532 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:56:00,534 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 41 msec 2023-03-28 20:56:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-28 20:56:00,597 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-28 20:56:00,611 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=232 (was 227) - Thread LEAK? -, OpenFileDescriptor=583 (was 563) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 365) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=6942 (was 6532) - AvailableMemoryMB LEAK? - 2023-03-28 20:56:00,624 INFO [Listener at localhost/34739] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=232, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=169, AvailableMemoryMB=6941 2023-03-28 20:56:00,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:56:00,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:56:00,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:00,628 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-28 20:56:00,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-28 20:56:00,628 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-28 20:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-28 20:56:00,629 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:00,630 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 empty. 2023-03-28 20:56:00,630 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:00,630 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:56:00,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-28 20:56:00,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-28 20:56:01,037 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:56:01,038 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1f0e2462e279947a10f94052c3152427, NAME => 'TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp 2023-03-28 20:56:01,042 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:56:01,042 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1f0e2462e279947a10f94052c3152427, disabling compactions & flushes 2023-03-28 20:56:01,042 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,042 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,043 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. after waiting 0 ms 2023-03-28 20:56:01,043 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,043 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,043 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:01,044 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-28 20:56:01,045 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036961045"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036961045"}]},"ts":"1680036961045"} 2023-03-28 20:56:01,046 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-28 20:56:01,046 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-28 20:56:01,047 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036961047"}]},"ts":"1680036961047"} 2023-03-28 20:56:01,047 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-28 20:56:01,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, ASSIGN}] 2023-03-28 20:56:01,052 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, ASSIGN 2023-03-28 20:56:01,052 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=false 2023-03-28 20:56:01,204 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:01,204 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036961203"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036961203"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036961203"}]},"ts":"1680036961203"} 2023-03-28 20:56:01,205 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:56:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-28 20:56:01,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1f0e2462e279947a10f94052c3152427, NAME => 'TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:56:01,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:56:01,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,361 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,362 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A 2023-03-28 20:56:01,362 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A 2023-03-28 20:56:01,362 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:01,362 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName A 2023-03-28 20:56:01,363 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:01,363 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,364 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B 2023-03-28 20:56:01,364 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B 2023-03-28 20:56:01,364 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:01,364 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName B 2023-03-28 20:56:01,364 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:01,365 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,365 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C 2023-03-28 20:56:01,366 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C 2023-03-28 20:56:01,366 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:01,366 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName C 2023-03-28 20:56:01,366 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:01,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:56:01,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:01,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-28 20:56:01,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1f0e2462e279947a10f94052c3152427; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74123117, jitterRate=0.1045205146074295}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:56:01,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:01,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., pid=56, masterSystemTime=1680036961356 2023-03-28 20:56:01,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:01,374 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:01,374 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036961373"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036961373"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036961373"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036961373"}]},"ts":"1680036961373"} 2023-03-28 20:56:01,375 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-28 20:56:01,376 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 in 169 msec 2023-03-28 20:56:01,377 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-28 20:56:01,377 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, ASSIGN in 326 msec 2023-03-28 20:56:01,377 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-28 20:56:01,377 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036961377"}]},"ts":"1680036961377"} 2023-03-28 20:56:01,378 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-28 20:56:01,381 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-28 20:56:01,382 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 755 msec 2023-03-28 20:56:01,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-28 20:56:01,733 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-28 20:56:01,733 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x0c0766f0 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:01,739 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b13e8aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:01,740 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:01,742 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51892, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:01,742 DEBUG [Listener at localhost/34739] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-28 20:56:01,745 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-28 20:56:01,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-28 20:56:01,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-28 20:56:01,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:02,159 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:56:02,160 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-28 20:56:02,160 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-28 20:56:02,162 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-28 20:56:02,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, REOPEN/MOVE}] 2023-03-28 20:56:02,165 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, REOPEN/MOVE 2023-03-28 20:56:02,165 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,165 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036962165"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036962165"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036962165"}]},"ts":"1680036962165"} 2023-03-28 20:56:02,166 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:56:02,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1f0e2462e279947a10f94052c3152427, disabling compactions & flushes 2023-03-28 20:56:02,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. after waiting 0 ms 2023-03-28 20:56:02,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-28 20:56:02,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:02,322 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1f0e2462e279947a10f94052c3152427 to self. 2023-03-28 20:56:02,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,324 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=CLOSED 2023-03-28 20:56:02,324 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036962324"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036962324"}]},"ts":"1680036962324"} 2023-03-28 20:56:02,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-28 20:56:02,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 in 159 msec 2023-03-28 20:56:02,326 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42693,1680036815956; forceNewPlan=false, retain=true 2023-03-28 20:56:02,477 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,477 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036962477"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036962477"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036962477"}]},"ts":"1680036962477"} 2023-03-28 20:56:02,478 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:56:02,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1f0e2462e279947a10f94052c3152427, NAME => 'TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.', STARTKEY => '', ENDKEY => ''} 2023-03-28 20:56:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-28 20:56:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,634 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,635 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A 2023-03-28 20:56:02,635 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A 2023-03-28 20:56:02,635 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:02,636 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName A 2023-03-28 20:56:02,637 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:02,637 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,638 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B 2023-03-28 20:56:02,638 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B 2023-03-28 20:56:02,638 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:02,638 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName B 2023-03-28 20:56:02,639 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:02,639 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,639 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C 2023-03-28 20:56:02,639 DEBUG [StoreOpener-1f0e2462e279947a10f94052c3152427-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C 2023-03-28 20:56:02,639 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-28 20:56:02,640 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f0e2462e279947a10f94052c3152427 columnFamilyName C 2023-03-28 20:56:02,640 INFO [StoreOpener-1f0e2462e279947a10f94052c3152427-1] regionserver.HStore(310): Store=1f0e2462e279947a10f94052c3152427/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-28 20:56:02,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-28 20:56:02,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1f0e2462e279947a10f94052c3152427; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73078927, jitterRate=0.08896087110042572}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-28 20:56:02,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:02,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., pid=61, masterSystemTime=1680036962630 2023-03-28 20:56:02,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,647 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,647 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036962647"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680036962647"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680036962647"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680036962647"}]},"ts":"1680036962647"} 2023-03-28 20:56:02,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-28 20:56:02,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 in 170 msec 2023-03-28 20:56:02,650 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-28 20:56:02,650 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, REOPEN/MOVE in 485 msec 2023-03-28 20:56:02,651 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-28 20:56:02,651 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-28 20:56:02,653 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-03-28 20:56:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-28 20:56:02,654 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x6257294f to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,659 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d0f313, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,659 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x52104714 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,663 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9d61148, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,664 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x4a320817 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,668 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a4acea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,669 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,672 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,673 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,676 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,677 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x3c1a68cd to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,681 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ea1da2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,681 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x5a709b21 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,686 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35c3cca2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,687 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x156e4ba6 to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,693 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b1973b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,693 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x785e366b to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,698 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0a71bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,698 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(139): Connect 0x5fdfef3b to 127.0.0.1:56827 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-28 20:56:02,702 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bb77c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-28 20:56:02,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:02,716 DEBUG [hconnection-0x1c443cd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,717 DEBUG [hconnection-0x586ea5cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,717 DEBUG [hconnection-0x2c480386-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,718 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51914, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,720 DEBUG [hconnection-0x6ffac691-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,720 DEBUG [hconnection-0x3f8eeb06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,723 DEBUG [hconnection-0x37c801c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,724 DEBUG [hconnection-0x40f47bc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,725 DEBUG [hconnection-0x90f9bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,726 DEBUG [hconnection-0x1646779e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:02,727 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,727 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51930, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,727 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,727 DEBUG [hconnection-0x39f9e47b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-28 20:56:02,728 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51924, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,728 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,728 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:02,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:02,729 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,731 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51940, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:02,731 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:02,731 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,735 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51962, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-28 20:56:02,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:02,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:02,746 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,747 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,747 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:02,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:02,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:02,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:02,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:02,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:02,748 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:02,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:02,752 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,752 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:02,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-28 20:56:02,753 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:02,753 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:02,753 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:02,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:02,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:02,755 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:02,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:02,768 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037022766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037022768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037022767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:02,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321d46a2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:02,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:02,771 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037022770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,772 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037022771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:02,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,773 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:02,774 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:02,777 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:02,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:02,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:02,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:02,777 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:02,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:02,780 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:02,780 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,782 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,785 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:02,785 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:02,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:02,785 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,786 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:02,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:02,786 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:02,786 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:02,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:02,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:02,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037022871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037022872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037022872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037022873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:02,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037022875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,886 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:02,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:02,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:02,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:02,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:02,890 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:02,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:02,890 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:02,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:02,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:02,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:02,892 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,893 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:02,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:02,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:02,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:02,894 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,894 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:02,895 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:02,895 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:02,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:02,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:02,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:02,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:02,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:02,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508c78b8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:02,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:02,898 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:02,903 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:02,905 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:02,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:02,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:02,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:02,905 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:02,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,907 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:02,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:02,909 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,910 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:02,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:02,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:02,912 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:02,912 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:02,913 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:02,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:02,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:02,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:02,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,013 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,016 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,016 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,017 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,017 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,019 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,019 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,020 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,020 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,021 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,021 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,021 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,021 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,026 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e9ceea[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:03,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:03,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:03,030 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:03,030 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:03,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:03,032 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:03,034 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:03,034 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:03,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:03,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:03,035 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:03,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,037 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,038 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:03,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:03,042 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:03,042 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:03,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:03,041 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037023073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037023074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037023075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037023076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037023079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,142 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,146 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,146 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,148 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,148 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,150 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,150 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,150 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,151 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:03,154 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2beab15b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:03,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:03,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,157 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:03,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:03,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:03,157 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,157 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:03,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:03,162 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:03,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:03,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:03,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:03,162 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:03,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,165 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,165 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,167 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:03,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,171 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,171 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:03,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,171 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,201 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328726f31594bfe4f879f0134a941c9e6fa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328726f31594bfe4f879f0134a941c9e6fa 2023-03-28 20:56:03,271 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,275 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,277 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,279 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,279 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,279 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:03,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739acad2[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:03,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:03,282 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:03,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:03,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:03,294 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:03,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:03,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:03,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:03,294 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:03,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:03,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:03,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:03,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,301 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:03,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,301 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:03,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037023378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037023378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037023379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037023380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037023384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,401 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,416 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,417 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,420 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:03,422 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,422 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e035a1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:03,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:03,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,433 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:03,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,433 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:03,435 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:03,435 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,435 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,435 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:03,437 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:03,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:03,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:03,437 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:03,437 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:03,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,439 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,439 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,440 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,442 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:03,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,443 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,443 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:03,443 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:03,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:03,543 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,547 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,550 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,551 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,552 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,552 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,552 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:03,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:03,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:03,552 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:03,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,553 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a443dc9[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:03,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:03,556 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:03,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:03,557 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:03,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:03,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:03,561 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:03,561 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:03,564 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,564 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,564 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:03,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:03,565 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,566 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:03,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:03,569 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,569 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:03,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:03,569 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:03,569 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:03,608 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1b709773445d4b18ba7b6e3c6cdcdb6e 2023-03-28 20:56:03,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3c062dd39da549bda10c8a19f217da89 2023-03-28 20:56:03,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/1395e3575e9a43ca9c1ed098561b9db5 2023-03-28 20:56:03,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1b709773445d4b18ba7b6e3c6cdcdb6e as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e 2023-03-28 20:56:03,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e, entries=150, sequenceid=17, filesize=25.1 K 2023-03-28 20:56:03,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3c062dd39da549bda10c8a19f217da89 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89 2023-03-28 20:56:03,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:56:03,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/1395e3575e9a43ca9c1ed098561b9db5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5 2023-03-28 20:56:03,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5, entries=150, sequenceid=17, filesize=11.6 K 2023-03-28 20:56:03,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1f0e2462e279947a10f94052c3152427 in 898ms, sequenceid=17, compaction requested=false 2023-03-28 20:56:03,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:03,669 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:03,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:03,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:03,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:03,673 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:03,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:03,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:03,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,674 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:03,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,675 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:03,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:03,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:03,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:03,681 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:03,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:03,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:03,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037023888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037023889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037023889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037023890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037023890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037023991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037023992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037023992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037023992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:03,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:03,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037023993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,095 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032819dcd1e0d226404e89dba211d512a6e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032819dcd1e0d226404e89dba211d512a6e7 2023-03-28 20:56:04,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037024195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037024195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037024195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037024195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037024196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037024497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037024498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037024498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037024499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,500 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/4c34fd1c687246ffb555af6a4c24f25b 2023-03-28 20:56:04,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037024499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,510 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/51bc650255b4468f8b195c0f42e9c726 2023-03-28 20:56:04,917 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/366044011eb547308b2b808cbc9d2319 2023-03-28 20:56:04,921 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/4c34fd1c687246ffb555af6a4c24f25b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b 2023-03-28 20:56:04,924 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b, entries=150, sequenceid=36, filesize=25.1 K 2023-03-28 20:56:04,925 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/51bc650255b4468f8b195c0f42e9c726 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726 2023-03-28 20:56:04,928 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726, entries=150, sequenceid=36, filesize=11.6 K 2023-03-28 20:56:04,929 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/366044011eb547308b2b808cbc9d2319 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319 2023-03-28 20:56:04,932 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319, entries=150, sequenceid=36, filesize=11.6 K 2023-03-28 20:56:04,933 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1f0e2462e279947a10f94052c3152427 in 1252ms, sequenceid=36, compaction requested=false 2023-03-28 20:56:04,933 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:04,933 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:04,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,935 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:04,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,935 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:04,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:04,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a3bb3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:04,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:04,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,938 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:04,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,939 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:04,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:04,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:04,943 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:04,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:04,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:04,943 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:04,943 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:04,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,944 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,945 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:04,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:04,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:04,945 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,947 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:04,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:04,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:04,950 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:04,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:04,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:04,949 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:04,950 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:04,950 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:04,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:05,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:05,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,015 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032886312aa4c6044e57801b2d436b9907f2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032886312aa4c6044e57801b2d436b9907f2 2023-03-28 20:56:05,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037025014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037025014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037025015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037025016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037025017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,020 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/2a02c11d603d4af68281096b124de5e0 2023-03-28 20:56:05,050 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,053 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,056 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,056 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,057 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,058 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,059 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,060 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,060 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2133adb0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,063 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,064 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,064 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,069 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,073 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,073 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,073 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,080 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,080 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037025119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037025119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037025120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037025120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037025121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,180 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,183 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,183 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,188 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,188 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,188 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,188 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,190 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603fc90a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,191 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,192 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,193 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,193 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,194 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,194 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,200 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,200 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,203 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,203 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,205 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,207 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,208 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,208 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,208 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:05,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,308 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,312 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,312 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,314 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,314 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,316 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,320 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,320 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,322 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,322 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037025322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,323 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037025322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bce0ab1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,324 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037025322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037025323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037025323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,326 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,327 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,327 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,328 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,328 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,331 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,331 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,331 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,333 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,333 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,334 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,337 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,337 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:05,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,338 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,338 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,338 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/9e658e981902415b8d9e59992c8802aa 2023-03-28 20:56:05,438 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,443 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,443 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,444 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,447 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,447 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,448 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,449 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,449 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,449 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,449 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,449 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,451 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,451 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9c39d8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,453 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,454 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,454 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,454 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,455 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,456 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,458 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,458 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,458 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,460 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,461 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,462 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,468 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,468 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,468 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,469 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,569 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,572 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,572 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,573 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,577 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,577 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,580 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,581 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52fe4796[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,584 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,586 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,589 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,589 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,591 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,593 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,595 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,596 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,596 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,596 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037025625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037025625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037025626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037025627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:05,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037025628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,643 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-28 20:56:05,696 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,700 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,701 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,702 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,702 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,703 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,703 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,720 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,728 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,728 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,728 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:05,728 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:05,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:05,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:05,728 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:05,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c97a35a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:05,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:05,733 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,734 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:05,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:05,735 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:05,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,735 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:05,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:05,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,737 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:05,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:05,739 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:05,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:05,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:05,739 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:05,739 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:05,741 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,741 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,741 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:05,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:05,743 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,744 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:05,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:05,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:05,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,748 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:05,748 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:05,749 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:05,749 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:05,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:05,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f108047291204698ad190e7d5a37cd52 2023-03-28 20:56:05,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/2a02c11d603d4af68281096b124de5e0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0 2023-03-28 20:56:05,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0, entries=150, sequenceid=50, filesize=25.1 K 2023-03-28 20:56:05,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/9e658e981902415b8d9e59992c8802aa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa 2023-03-28 20:56:05,849 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:05,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:05,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa, entries=150, sequenceid=50, filesize=11.6 K 2023-03-28 20:56:05,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:05,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:05,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f108047291204698ad190e7d5a37cd52 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52 2023-03-28 20:56:05,854 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:05,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:05,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:05,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,854 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,856 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:05,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:05,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:05,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:05,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:05,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52, entries=150, sequenceid=50, filesize=11.6 K 2023-03-28 20:56:05,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:05,857 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-28 20:56:05,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1f0e2462e279947a10f94052c3152427 in 856ms, sequenceid=50, compaction requested=true 2023-03-28 20:56:05,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:05,858 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:05,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:05,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:05,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:05,859 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:05,859 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:05,859 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:05,859 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:05,859 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:05,860 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,860 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,860 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=75.4 K 2023-03-28 20:56:05,860 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=34.8 K 2023-03-28 20:56:05,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:05,860 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1b709773445d4b18ba7b6e3c6cdcdb6e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036962744 2023-03-28 20:56:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:05,860 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3c062dd39da549bda10c8a19f217da89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036962744 2023-03-28 20:56:05,860 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4c34fd1c687246ffb555af6a4c24f25b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680036962759 2023-03-28 20:56:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:05,861 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:05,860 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:05,861 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:56:05,861 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 51bc650255b4468f8b195c0f42e9c726, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680036962759 2023-03-28 20:56:05,861 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 2a02c11d603d4af68281096b124de5e0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:05,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:05,861 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 9e658e981902415b8d9e59992c8802aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:05,881 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:05,905 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#565 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:05,908 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/b70bc470870f40c899aebe91a1a1c20f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b70bc470870f40c899aebe91a1a1c20f 2023-03-28 20:56:05,912 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into b70bc470870f40c899aebe91a1a1c20f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:05,912 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:05,912 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=13, startTime=1680036965859; duration=0sec 2023-03-28 20:56:05,913 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:05,913 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:05,913 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:05,913 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:05,914 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:05,914 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=34.8 K 2023-03-28 20:56:05,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1395e3575e9a43ca9c1ed098561b9db5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680036962744 2023-03-28 20:56:05,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 366044011eb547308b2b808cbc9d2319, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680036962759 2023-03-28 20:56:05,914 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f108047291204698ad190e7d5a37cd52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:05,921 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#567 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:06,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:06,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:06,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037026137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037026137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037026138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037026140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037026140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037026241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037026241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037026241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037026242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037026244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,297 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328ea14b168476e46a88e114958b16a19dd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ea14b168476e46a88e114958b16a19dd 2023-03-28 20:56:06,304 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/34b885675e374e578e3cead37332a90c 2023-03-28 20:56:06,351 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/54cded68b6874625b34caa901105e4be as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/54cded68b6874625b34caa901105e4be 2023-03-28 20:56:06,354 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into 54cded68b6874625b34caa901105e4be(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:06,354 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:06,355 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=13, startTime=1680036965859; duration=0sec 2023-03-28 20:56:06,355 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:06,358 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/c02f4c62e7ff46639cc2986912fd02d2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/c02f4c62e7ff46639cc2986912fd02d2 2023-03-28 20:56:06,361 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into c02f4c62e7ff46639cc2986912fd02d2(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:06,361 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:06,362 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=13, startTime=1680036965858; duration=0sec 2023-03-28 20:56:06,362 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:06,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037026444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037026444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037026444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037026445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037026447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,714 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/6ad91565a0ce4052b6f18fc38db8dc1b 2023-03-28 20:56:06,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037026746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037026747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037026749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037026749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037026750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,129 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/e9527b29ef8149ce81131aa7dba01714 2023-03-28 20:56:07,133 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/34b885675e374e578e3cead37332a90c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c 2023-03-28 20:56:07,136 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c, entries=150, sequenceid=68, filesize=25.1 K 2023-03-28 20:56:07,137 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/6ad91565a0ce4052b6f18fc38db8dc1b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b 2023-03-28 20:56:07,140 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b, entries=150, sequenceid=68, filesize=11.6 K 2023-03-28 20:56:07,140 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/e9527b29ef8149ce81131aa7dba01714 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714 2023-03-28 20:56:07,144 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714, entries=150, sequenceid=68, filesize=11.6 K 2023-03-28 20:56:07,145 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1f0e2462e279947a10f94052c3152427 in 1284ms, sequenceid=68, compaction requested=false 2023-03-28 20:56:07,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:07,145 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:07,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:07,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:07,145 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:07,147 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,147 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,148 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:07,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7b3fb[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:07,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:07,149 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,150 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:07,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:07,151 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:07,153 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:07,153 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,153 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:07,153 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:07,156 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:07,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:07,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:07,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:07,156 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:07,158 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,158 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,159 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,160 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:07,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:07,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:07,162 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:07,162 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,162 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,162 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:07,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:07,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,262 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:07,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:07,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:07,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:07,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:07,267 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:07,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:07,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:07,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037027263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,268 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303283738966fd9184a129b9936bc0b1d5639 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303283738966fd9184a129b9936bc0b1d5639 2023-03-28 20:56:07,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,270 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037027266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,271 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037027267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037027266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037027268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:07,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:07,272 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:07,272 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,272 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:07,273 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,273 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:07,273 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:07,273 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:07,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:07,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,275 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,275 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@81a0822[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:07,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:07,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,279 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:07,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:07,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:07,279 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:07,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:07,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,281 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,281 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:07,283 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:07,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:07,283 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:07,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:07,284 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:07,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,286 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,288 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,291 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:07,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,291 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,291 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:07,291 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:07,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:07,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037027368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037027372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037027372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037027372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037027372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,391 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:07,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:07,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:07,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:07,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:07,395 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,399 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,399 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:07,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:07,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:07,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:07,400 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:07,400 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,400 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:07,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:07,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:07,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:07,401 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:07,403 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,403 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,403 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,403 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28783c40[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:07,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:07,405 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,407 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,407 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:07,407 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:07,407 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:07,407 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,407 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:07,408 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,408 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:07,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:07,412 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:07,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:07,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:07,412 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:07,412 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:07,414 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,414 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,414 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:07,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:07,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,418 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,418 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,418 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:07,419 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:07,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,519 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:07,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:07,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:07,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:07,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:07,522 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,525 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,525 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:07,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:07,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:07,527 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,527 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:07,527 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:07,527 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:07,529 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,530 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5904d766[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:07,531 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:07,531 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,532 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:07,534 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:07,536 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:07,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:07,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:07,536 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:07,536 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:07,539 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,539 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,539 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,540 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,541 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,543 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:07,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,543 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,543 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:07,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037027571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037027575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037027575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037027576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037027576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,643 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:07,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:07,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:07,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:07,647 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,651 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,651 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,652 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:07,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:07,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:07,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:07,653 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,653 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:07,653 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:07,653 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,653 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:07,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:07,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:07,654 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:07,655 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,655 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,655 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,656 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:07,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e6efd6[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:07,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:07,657 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,659 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:07,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,659 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,661 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:07,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:07,664 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:07,664 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:07,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,666 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,668 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,674 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=88, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/9cd37a7fbd614503b45570dee8720c3c 2023-03-28 20:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,683 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:07,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:07,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:07,683 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:07,684 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:07,684 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:07,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:07,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/11f291b714e04c39a05616e4ce0f8242 2023-03-28 20:56:07,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f5683deab72c4e1a83e5f53f7523c2f4 2023-03-28 20:56:07,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/9cd37a7fbd614503b45570dee8720c3c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c 2023-03-28 20:56:07,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c, entries=150, sequenceid=88, filesize=25.1 K 2023-03-28 20:56:07,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/11f291b714e04c39a05616e4ce0f8242 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242 2023-03-28 20:56:07,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242, entries=150, sequenceid=88, filesize=11.6 K 2023-03-28 20:56:07,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f5683deab72c4e1a83e5f53f7523c2f4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4 2023-03-28 20:56:07,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4, entries=150, sequenceid=88, filesize=11.6 K 2023-03-28 20:56:07,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1f0e2462e279947a10f94052c3152427 in 483ms, sequenceid=88, compaction requested=true 2023-03-28 20:56:07,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:07,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:07,737 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:07,737 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:07,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:07,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:07,738 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:07,738 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:07,738 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:07,738 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,738 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b70bc470870f40c899aebe91a1a1c20f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=34.9 K 2023-03-28 20:56:07,738 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:07,739 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b70bc470870f40c899aebe91a1a1c20f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:07,739 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,739 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/c02f4c62e7ff46639cc2986912fd02d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=75.5 K 2023-03-28 20:56:07,739 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6ad91565a0ce4052b6f18fc38db8dc1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680036965012 2023-03-28 20:56:07,739 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c02f4c62e7ff46639cc2986912fd02d2, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:07,740 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 11f291b714e04c39a05616e4ce0f8242, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:07,740 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 34b885675e374e578e3cead37332a90c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680036965012 2023-03-28 20:56:07,740 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 9cd37a7fbd614503b45570dee8720c3c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:07,746 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:07,762 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#574 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:07,784 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:07,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:07,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:07,787 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:07,787 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,787 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:07,787 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:07,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,789 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:07,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:07,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:07,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:07,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:07,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:07,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:07,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:07,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:07,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:07,790 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:07,790 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:07,791 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:07,791 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:07,791 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:56:07,791 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:07,792 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,792 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:07,792 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,792 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:07,792 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:07,817 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328af7c5ecaaf324f9aa6c6d041bf4c7a0d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328af7c5ecaaf324f9aa6c6d041bf4c7a0d 2023-03-28 20:56:07,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:07,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:07,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037027888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037027910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037027910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037027911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:07,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037027914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037028012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037028016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037028017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037028017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037028017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,163 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f28ce113aa1147a389aeab60417793c5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f28ce113aa1147a389aeab60417793c5 2023-03-28 20:56:08,167 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into f28ce113aa1147a389aeab60417793c5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:08,167 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:08,167 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=13, startTime=1680036967737; duration=0sec 2023-03-28 20:56:08,167 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:08,167 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:08,168 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:08,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:08,169 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:08,169 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/54cded68b6874625b34caa901105e4be, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=34.9 K 2023-03-28 20:56:08,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 54cded68b6874625b34caa901105e4be, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680036963888 2023-03-28 20:56:08,169 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e9527b29ef8149ce81131aa7dba01714, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680036965012 2023-03-28 20:56:08,170 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f5683deab72c4e1a83e5f53f7523c2f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:08,176 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#576 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:08,181 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e4b3fdfd52c743b9bccfd1da1b390175 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e4b3fdfd52c743b9bccfd1da1b390175 2023-03-28 20:56:08,186 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/b24d570d2d734d1dbfd1898bb5ee1104 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b24d570d2d734d1dbfd1898bb5ee1104 2023-03-28 20:56:08,186 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into e4b3fdfd52c743b9bccfd1da1b390175(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:08,186 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:08,186 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=13, startTime=1680036967737; duration=0sec 2023-03-28 20:56:08,187 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:08,191 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into b24d570d2d734d1dbfd1898bb5ee1104(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:08,191 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:08,191 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=13, startTime=1680036967737; duration=0sec 2023-03-28 20:56:08,191 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:08,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037028214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037028219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037028220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037028220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037028220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,228 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/b73c9ca65c6b45629ad5897e3a62bbc5 2023-03-28 20:56:08,238 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/bca7d690824a4d3185d1d3c21a7af274 2023-03-28 20:56:08,247 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/a2ec3ff90900482eb89aabb93c6c0c13 2023-03-28 20:56:08,250 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/b73c9ca65c6b45629ad5897e3a62bbc5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5 2023-03-28 20:56:08,253 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5, entries=150, sequenceid=103, filesize=25.1 K 2023-03-28 20:56:08,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/bca7d690824a4d3185d1d3c21a7af274 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274 2023-03-28 20:56:08,257 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274, entries=150, sequenceid=103, filesize=11.6 K 2023-03-28 20:56:08,258 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/a2ec3ff90900482eb89aabb93c6c0c13 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13 2023-03-28 20:56:08,261 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13, entries=150, sequenceid=103, filesize=11.6 K 2023-03-28 20:56:08,261 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 1f0e2462e279947a10f94052c3152427 in 470ms, sequenceid=103, compaction requested=false 2023-03-28 20:56:08,262 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:08,262 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:08,264 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:08,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b65b090[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:08,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:08,267 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,268 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:08,268 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,268 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:08,269 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:08,272 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:08,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:08,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:08,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:08,272 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:08,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:08,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:08,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:08,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,275 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:08,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,276 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:08,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:08,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:08,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:08,280 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:08,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:08,281 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,281 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:08,281 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:08,381 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:08,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:08,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:08,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:08,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:08,384 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:08,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:08,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:08,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:08,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,385 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:08,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:08,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:08,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:08,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,386 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:08,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:08,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:08,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:08,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:08,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:08,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:08,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:08,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:08,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:08,387 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:08,387 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:08,387 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:08,388 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:08,388 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:08,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:08,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:08,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037028529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037028529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037028554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037028554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037028554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037028655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037028655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037028658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037028658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037028658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,813 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032875eb5096b5314d6a93ac522bb44caaa0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875eb5096b5314d6a93ac522bb44caaa0 2023-03-28 20:56:08,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037028858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037028859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037028861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037028863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:08,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:08,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037028863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037029161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037029162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037029166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037029166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037029167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,229 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/549d54bf8eb7450ba2be864f1b6ba64d 2023-03-28 20:56:09,251 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f2c822d0bfdb4caba9472c7847b14d06 2023-03-28 20:56:09,266 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/9c7c0c7f1ab1427097499f29caf7bdd0 2023-03-28 20:56:09,272 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/549d54bf8eb7450ba2be864f1b6ba64d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d 2023-03-28 20:56:09,278 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d, entries=150, sequenceid=124, filesize=25.1 K 2023-03-28 20:56:09,279 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f2c822d0bfdb4caba9472c7847b14d06 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06 2023-03-28 20:56:09,283 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06, entries=150, sequenceid=124, filesize=11.6 K 2023-03-28 20:56:09,283 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/9c7c0c7f1ab1427097499f29caf7bdd0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0 2023-03-28 20:56:09,289 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0, entries=150, sequenceid=124, filesize=11.6 K 2023-03-28 20:56:09,290 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 1f0e2462e279947a10f94052c3152427 in 902ms, sequenceid=124, compaction requested=true 2023-03-28 20:56:09,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:09,290 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:09,290 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:09,292 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,292 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:09,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:09,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:09,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52961180[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:09,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:09,294 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,296 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:09,297 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:09,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:09,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:09,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:09,301 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:09,301 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:09,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:09,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:09,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:09,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,306 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:09,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:09,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:09,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:09,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:09,309 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:09,309 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:09,310 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:09,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:09,410 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:09,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:09,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:09,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:09,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:09,413 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:09,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:09,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:09,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:09,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:09,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:09,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:09,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:09,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,416 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:09,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:09,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:09,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:09,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:09,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:09,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:09,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:09,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:09,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:09,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:09,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:09,417 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:56:09,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:09,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:09,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:09,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:09,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:09,418 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:09,430 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303285e4d411bf70340ecb9ed0286b7c50ada to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e4d411bf70340ecb9ed0286b7c50ada 2023-03-28 20:56:09,436 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1987eead72504e8fb2220c09803e7e39 2023-03-28 20:56:09,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:09,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:09,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037029690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037029691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037029691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037029692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037029693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037029794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037029795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037029796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037029796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037029796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:09,847 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/6994cbe1506c44a6a71416168e8f847a 2023-03-28 20:56:09,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:09,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037029998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037029999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037029999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037029999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037030000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,257 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/b165cfd346f142a9adc084ad38240321 2023-03-28 20:56:10,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1987eead72504e8fb2220c09803e7e39 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39 2023-03-28 20:56:10,265 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39, entries=150, sequenceid=139, filesize=25.3 K 2023-03-28 20:56:10,266 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/6994cbe1506c44a6a71416168e8f847a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a 2023-03-28 20:56:10,269 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a, entries=150, sequenceid=139, filesize=11.8 K 2023-03-28 20:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/b165cfd346f142a9adc084ad38240321 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321 2023-03-28 20:56:10,273 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321, entries=150, sequenceid=139, filesize=11.8 K 2023-03-28 20:56:10,273 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1f0e2462e279947a10f94052c3152427 in 856ms, sequenceid=139, compaction requested=true 2023-03-28 20:56:10,274 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:10,274 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:10,276 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d564353[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:10,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:10,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,287 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:10,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,287 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:10,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:10,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:10,289 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:10,292 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:10,292 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:10,294 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,294 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:10,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:10,294 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:10,295 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,296 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,299 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:10,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:10,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:10,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:10,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:10,299 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:10,299 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,299 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:10,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:10,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:10,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:10,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,315 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328fb069533665b426297aadd5ab7234938 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328fb069533665b426297aadd5ab7234938 2023-03-28 20:56:10,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037030312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037030312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037030313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037030315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037030315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,320 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/69f5e73143384613b03bf64cb4a88407 2023-03-28 20:56:10,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3676aa396e7c49db87278be5b614a2e2 2023-03-28 20:56:10,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/c064323e615a4ab088e70de72085bb6b 2023-03-28 20:56:10,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/69f5e73143384613b03bf64cb4a88407 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407 2023-03-28 20:56:10,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407, entries=150, sequenceid=157, filesize=25.3 K 2023-03-28 20:56:10,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3676aa396e7c49db87278be5b614a2e2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2 2023-03-28 20:56:10,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2, entries=150, sequenceid=157, filesize=11.8 K 2023-03-28 20:56:10,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/c064323e615a4ab088e70de72085bb6b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b 2023-03-28 20:56:10,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b, entries=150, sequenceid=157, filesize=11.8 K 2023-03-28 20:56:10,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 1f0e2462e279947a10f94052c3152427 in 58ms, sequenceid=157, compaction requested=true 2023-03-28 20:56:10,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:10,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:10,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:10,362 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:56:10,362 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:56:10,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:10,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:56:10,364 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:10,364 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129236 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:56:10,364 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:10,364 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:10,364 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f28ce113aa1147a389aeab60417793c5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=58.5 K 2023-03-28 20:56:10,364 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:10,365 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e4b3fdfd52c743b9bccfd1da1b390175, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=126.2 K 2023-03-28 20:56:10,365 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f28ce113aa1147a389aeab60417793c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:10,365 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting bca7d690824a4d3185d1d3c21a7af274, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680036967264 2023-03-28 20:56:10,366 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e4b3fdfd52c743b9bccfd1da1b390175, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:10,366 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f2c822d0bfdb4caba9472c7847b14d06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680036967891 2023-03-28 20:56:10,366 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b73c9ca65c6b45629ad5897e3a62bbc5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680036967264 2023-03-28 20:56:10,367 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 549d54bf8eb7450ba2be864f1b6ba64d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680036967891 2023-03-28 20:56:10,367 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 6994cbe1506c44a6a71416168e8f847a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680036968527 2023-03-28 20:56:10,367 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1987eead72504e8fb2220c09803e7e39, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680036968527 2023-03-28 20:56:10,367 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3676aa396e7c49db87278be5b614a2e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:10,368 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 69f5e73143384613b03bf64cb4a88407, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:10,380 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:10,382 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#588 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:10,399 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:10,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:10,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:10,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:10,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:10,409 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:10,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,410 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:10,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,411 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:10,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:10,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,411 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:10,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:10,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:10,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:10,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:10,413 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:10,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:10,414 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:10,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:10,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:10,416 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:10,416 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:10,416 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e1be5c961e1844afba8a2768145c63e9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e1be5c961e1844afba8a2768145c63e9 2023-03-28 20:56:10,417 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:10,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:10,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:10,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:10,421 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into e1be5c961e1844afba8a2768145c63e9(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:10,421 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:10,421 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=11, startTime=1680036970362; duration=0sec 2023-03-28 20:56:10,422 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:10,422 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-28 20:56:10,423 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-28 20:56:10,423 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:10,423 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:10,423 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b24d570d2d734d1dbfd1898bb5ee1104, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=58.5 K 2023-03-28 20:56:10,423 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b24d570d2d734d1dbfd1898bb5ee1104, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680036966137 2023-03-28 20:56:10,424 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting a2ec3ff90900482eb89aabb93c6c0c13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680036967264 2023-03-28 20:56:10,424 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 9c7c0c7f1ab1427097499f29caf7bdd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680036967891 2023-03-28 20:56:10,424 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b165cfd346f142a9adc084ad38240321, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1680036968527 2023-03-28 20:56:10,425 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting c064323e615a4ab088e70de72085bb6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:10,437 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#591 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:10,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037030433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037030433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037030436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037030437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037030438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,442 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328f470450766dc43aa9576370793fc6b10 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f470450766dc43aa9576370793fc6b10 2023-03-28 20:56:10,459 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/bae6277df83b4980b7f42f3ec0ef9269 2023-03-28 20:56:10,462 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/66c914b3a8fb4bf5aa5f78b3cb842960 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/66c914b3a8fb4bf5aa5f78b3cb842960 2023-03-28 20:56:10,468 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into 66c914b3a8fb4bf5aa5f78b3cb842960(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:10,468 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:10,468 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=11, startTime=1680036970362; duration=0sec 2023-03-28 20:56:10,468 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037030540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037030540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037030541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037030541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037030542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037030743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037030743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037030743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037030744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037030744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:10,800 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/5952d8f0e433486992fc92822aea7a4a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5952d8f0e433486992fc92822aea7a4a 2023-03-28 20:56:10,803 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into 5952d8f0e433486992fc92822aea7a4a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:10,803 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:10,803 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=11, startTime=1680036970362; duration=0sec 2023-03-28 20:56:10,804 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:10,874 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/7268b624ce384033884bbe9d5dcad31a 2023-03-28 20:56:11,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037031047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037031047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037031047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037031049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037031049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,284 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/1763992fe52e4155b144eafbefe17887 2023-03-28 20:56:11,287 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/bae6277df83b4980b7f42f3ec0ef9269 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269 2023-03-28 20:56:11,291 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269, entries=150, sequenceid=172, filesize=25.3 K 2023-03-28 20:56:11,291 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/7268b624ce384033884bbe9d5dcad31a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a 2023-03-28 20:56:11,294 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a, entries=150, sequenceid=172, filesize=11.8 K 2023-03-28 20:56:11,295 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/1763992fe52e4155b144eafbefe17887 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887 2023-03-28 20:56:11,298 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887, entries=150, sequenceid=172, filesize=11.8 K 2023-03-28 20:56:11,299 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1f0e2462e279947a10f94052c3152427 in 882ms, sequenceid=172, compaction requested=false 2023-03-28 20:56:11,299 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:11,299 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:11,301 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,301 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:11,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791a8170[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:11,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:11,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,304 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:11,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:11,305 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:11,307 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,307 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:11,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:11,310 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:11,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:11,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:11,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:11,310 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:11,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:11,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,312 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,315 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,317 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:11,318 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:11,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:11,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:11,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:11,318 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:11,318 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:11,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:11,418 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:11,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:11,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:11,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:11,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:11,421 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:11,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:11,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:11,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:11,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,422 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,424 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:11,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:11,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:11,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:11,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:11,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:11,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:11,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:11,425 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:11,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:11,426 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:11,426 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:11,426 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:11,427 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:11,427 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:11,442 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303282cfb568fd1094de4b23dcf318f697f6d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282cfb568fd1094de4b23dcf318f697f6d 2023-03-28 20:56:11,447 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/4043c8451a884496b439dac9ba5e992d 2023-03-28 20:56:11,461 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/e63d14e21ae847da90a19aaa071aec1c 2023-03-28 20:56:11,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:11,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:11,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037031559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037031559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037031560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037031561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037031562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037031663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037031663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037031665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037031665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037031666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037031865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037031866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037031867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037031868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:11,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037031869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,874 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/92193ad7c9e644cea5e280452df5be40 2023-03-28 20:56:11,877 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/4043c8451a884496b439dac9ba5e992d as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d 2023-03-28 20:56:11,881 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d, entries=150, sequenceid=192, filesize=25.3 K 2023-03-28 20:56:11,882 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/e63d14e21ae847da90a19aaa071aec1c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c 2023-03-28 20:56:11,885 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c, entries=150, sequenceid=192, filesize=11.8 K 2023-03-28 20:56:11,886 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/92193ad7c9e644cea5e280452df5be40 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40 2023-03-28 20:56:11,889 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40, entries=150, sequenceid=192, filesize=11.8 K 2023-03-28 20:56:11,889 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 1f0e2462e279947a10f94052c3152427 in 463ms, sequenceid=192, compaction requested=true 2023-03-28 20:56:11,889 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:11,889 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:11,889 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:11,890 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fed4ff3[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:11,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:11,894 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,895 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:11,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,895 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:11,896 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:11,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:11,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:11,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:11,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:11,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:11,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:11,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:11,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,902 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,905 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:11,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:11,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:11,905 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:11,906 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:11,906 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:11,906 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:11,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:11,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:12,006 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:12,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:12,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:12,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:12,010 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:12,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:12,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:12,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:12,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,010 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,012 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:12,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:12,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,012 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:12,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:12,014 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:12,014 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:12,014 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:12,029 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303287879ec273f964082aff23aa1a5e5031e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287879ec273f964082aff23aa1a5e5031e 2023-03-28 20:56:12,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:12,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:12,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037032181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037032182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037032183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037032185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037032186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037032287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037032287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037032287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037032291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037032291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,433 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=208, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/33cb7d88de944b549b28db32cc694613 2023-03-28 20:56:12,446 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/28d171aea4ea4996ba1bba7f4cb9725b 2023-03-28 20:56:12,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037032491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037032491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037032492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037032493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037032495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51956 deadline: 1680037032794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51896 deadline: 1680037032794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51962 deadline: 1680037032797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51924 deadline: 1680037032798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:12,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51940 deadline: 1680037032798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,854 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/cd929509cbe24fbb8d9dc467a5bb9bce 2023-03-28 20:56:12,857 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/33cb7d88de944b549b28db32cc694613 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613 2023-03-28 20:56:12,860 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613, entries=150, sequenceid=208, filesize=25.3 K 2023-03-28 20:56:12,861 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/28d171aea4ea4996ba1bba7f4cb9725b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b 2023-03-28 20:56:12,865 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b, entries=150, sequenceid=208, filesize=11.8 K 2023-03-28 20:56:12,865 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/cd929509cbe24fbb8d9dc467a5bb9bce as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce 2023-03-28 20:56:12,872 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce, entries=150, sequenceid=208, filesize=11.8 K 2023-03-28 20:56:12,873 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 1f0e2462e279947a10f94052c3152427 in 859ms, sequenceid=208, compaction requested=true 2023-03-28 20:56:12,873 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:12,873 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:12,874 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d15599f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:12,876 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:12,877 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,878 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:12,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,878 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:12,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:12,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:12,883 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,883 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:12,885 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:12,885 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:12,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:12,886 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:12,886 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:12,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:12,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,887 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:12,888 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,889 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:12,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:12,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:12,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:12,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:12,892 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,891 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:12,892 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:12,892 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:12,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:12,891 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:12,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:12,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:12,992 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:12,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:12,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:12,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:12,996 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:12,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:12,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:12,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:12,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:12,996 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,999 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:12,999 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:12,999 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:12,999 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:12,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:13,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:13,001 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,016 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303287fe6efddf5554d2f9df2f9de7b997523 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287fe6efddf5554d2f9df2f9de7b997523 2023-03-28 20:56:13,021 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=225, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e68d9bcac60a4c26a578f291d812b372 2023-03-28 20:56:13,033 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/95790a13384b41b4b710f6e3ff9f8cf9 2023-03-28 20:56:13,045 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/e5be5ac46dca4e82ac448bb85f99b658 2023-03-28 20:56:13,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e68d9bcac60a4c26a578f291d812b372 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372 2023-03-28 20:56:13,052 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372, entries=150, sequenceid=225, filesize=25.3 K 2023-03-28 20:56:13,053 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/95790a13384b41b4b710f6e3ff9f8cf9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9 2023-03-28 20:56:13,056 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9, entries=150, sequenceid=225, filesize=11.8 K 2023-03-28 20:56:13,057 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/e5be5ac46dca4e82ac448bb85f99b658 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658 2023-03-28 20:56:13,060 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658, entries=150, sequenceid=225, filesize=11.8 K 2023-03-28 20:56:13,061 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 1f0e2462e279947a10f94052c3152427 in 60ms, sequenceid=225, compaction requested=true 2023-03-28 20:56:13,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:13,061 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:13,063 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c6cfb5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:13,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:13,065 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,066 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:13,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,066 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:13,069 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:13,072 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:13,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:13,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:13,072 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:13,072 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:13,074 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,075 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,076 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,076 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,076 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,080 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:13,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:13,081 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,081 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:13,181 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:13,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:13,184 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:13,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:13,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:13,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,185 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,186 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,186 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,187 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:13,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:13,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:13,188 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,189 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:13,189 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:13,189 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:13,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:13,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:13,189 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:13,190 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,191 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffacd9c[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:13,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:13,193 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,195 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:13,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,195 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:13,196 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:13,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:13,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:13,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:13,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:13,199 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:13,199 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:13,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,201 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,203 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,206 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:13,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,206 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:13,207 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:13,206 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,207 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,307 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:13,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:13,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:13,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:13,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:13,310 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:13,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:13,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:13,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,311 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,313 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,313 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:13,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:13,314 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:13,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:13,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:13,315 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:13,315 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,315 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:13,316 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:13,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:13,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:13,330 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303282ddaae16bcef4196bcb9d891c7a24008 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282ddaae16bcef4196bcb9d891c7a24008 2023-03-28 20:56:13,337 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/7e5076b9c3514eec8fa9dddf84b22157 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036973358, max timestamp=1680036973363Num uniques 859; 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036973358, max timestamp=1680036973363Num uniques 859; 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680036973358, max timestamp=1680036973363Num uniques 859; 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:56:13,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-28 20:56:13,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680036973376, max timestamp=1680036973384Num uniques 1018; 2023-03-28 20:56:13,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-28 20:56:13,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680036973376, max timestamp=1680036973384Num uniques 1018; 2023-03-28 20:56:13,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-28 20:56:13,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680036973376, max timestamp=1680036973384Num uniques 1018; 2023-03-28 20:56:13,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-28 20:56:13,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036973397, max timestamp=1680036973402Num uniques 1127; 2023-03-28 20:56:13,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-28 20:56:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036973397, max timestamp=1680036973402Num uniques 1127; 2023-03-28 20:56:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-28 20:56:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680036973397, max timestamp=1680036973402Num uniques 1127; 2023-03-28 20:56:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-28 20:56:13,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680036973424, max timestamp=1680036973431Num uniques 1086; 2023-03-28 20:56:13,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680036973424, max timestamp=1680036973431Num uniques 1086; 2023-03-28 20:56:13,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-28 20:56:13,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-28 20:56:13,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680036973424, max timestamp=1680036973431Num uniques 1086; 2023-03-28 20:56:13,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-28 20:56:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680036973431, max timestamp=1680036973452Num uniques 1095; 2023-03-28 20:56:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=8920, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-28 20:56:13,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8929, totalHeapSize=20520, min timestamp=1680036973431, max timestamp=1680036973452Num uniques 1095; 2023-03-28 20:56:13,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50149, new segment data size=8929, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-28 20:56:13,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8929, totalHeapSize=20520, min timestamp=1680036973431, max timestamp=1680036973452Num uniques 1095; 2023-03-28 20:56:13,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50149, new segment data size=8929, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-28 20:56:13,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1680036973486, max timestamp=1680036973493Num uniques 1054; 2023-03-28 20:56:13,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=11624, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-28 20:56:13,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680036973489, max timestamp=1680036973493Num uniques 1054; 2023-03-28 20:56:13,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-28 20:56:13,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680036973489, max timestamp=1680036973493Num uniques 1054; 2023-03-28 20:56:13,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-28 20:56:13,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036973503, max timestamp=1680036973511Num uniques 1013; 2023-03-28 20:56:13,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-28 20:56:13,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036973503, max timestamp=1680036973511Num uniques 1113; 2023-03-28 20:56:13,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680036973503, max timestamp=1680036973511Num uniques 1013; 2023-03-28 20:56:13,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-28 20:56:13,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-28 20:56:13,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036973524, max timestamp=1680036973530Num uniques 1072; 2023-03-28 20:56:13,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036973524, max timestamp=1680036973530Num uniques 1072; 2023-03-28 20:56:13,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680036973524, max timestamp=1680036973530Num uniques 1072; 2023-03-28 20:56:13,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:56:13,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:56:13,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-28 20:56:13,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036973547, max timestamp=1680036973554Num uniques 1081; 2023-03-28 20:56:13,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:56:13,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036973547, max timestamp=1680036973554Num uniques 1081; 2023-03-28 20:56:13,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:56:13,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680036973547, max timestamp=1680036973554Num uniques 1081; 2023-03-28 20:56:13,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-28 20:56:13,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680036973562, max timestamp=1680036973573Num uniques 1040; 2023-03-28 20:56:13,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-03-28 20:56:13,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680036973562, max timestamp=1680036973573Num uniques 1040; 2023-03-28 20:56:13,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680036973562, max timestamp=1680036973573Num uniques 1040; 2023-03-28 20:56:13,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-03-28 20:56:13,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-03-28 20:56:13,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037033591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037033591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680036973586, max timestamp=1680036973594Num uniques 1249; 2023-03-28 20:56:13,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-28 20:56:13,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680036973586, max timestamp=1680036973594Num uniques 1249; 2023-03-28 20:56:13,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-28 20:56:13,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680036973586, max timestamp=1680036973594Num uniques 1249; 2023-03-28 20:56:13,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-28 20:56:13,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037033622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037033623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680036973620, max timestamp=1680036973626Num uniques 1158; 2023-03-28 20:56:13,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-28 20:56:13,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680036973620, max timestamp=1680036973626Num uniques 1158; 2023-03-28 20:56:13,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680036973620, max timestamp=1680036973626Num uniques 1158; 2023-03-28 20:56:13,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-28 20:56:13,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-28 20:56:13,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680036973686, max timestamp=1680036973698Num uniques 1017; 2023-03-28 20:56:13,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-28 20:56:13,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680036973686, max timestamp=1680036973698Num uniques 1017; 2023-03-28 20:56:13,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-28 20:56:13,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680036973686, max timestamp=1680036973698Num uniques 1017; 2023-03-28 20:56:13,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036973717, max timestamp=1680036973726Num uniques 1026; 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036973717, max timestamp=1680036973726Num uniques 1026; 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680036973717, max timestamp=1680036973726Num uniques 1026; 2023-03-28 20:56:13,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-28 20:56:13,744 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f0c684b71c2b4d449cdf842164b539dc 2023-03-28 20:56:13,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680036973736, max timestamp=1680036973745Num uniques 1035; 2023-03-28 20:56:13,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-28 20:56:13,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680036973736, max timestamp=1680036973745Num uniques 1035; 2023-03-28 20:56:13,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-28 20:56:13,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680036973736, max timestamp=1680036973745Num uniques 1035; 2023-03-28 20:56:13,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-28 20:56:13,753 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/80bc338d20cf4c9491dd9689f99580ea 2023-03-28 20:56:13,757 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/7e5076b9c3514eec8fa9dddf84b22157 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157 2023-03-28 20:56:13,760 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157, entries=150, sequenceid=232, filesize=25.3 K 2023-03-28 20:56:13,761 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f0c684b71c2b4d449cdf842164b539dc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc 2023-03-28 20:56:13,764 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc, entries=100, sequenceid=232, filesize=9.4 K 2023-03-28 20:56:13,765 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/80bc338d20cf4c9491dd9689f99580ea as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea 2023-03-28 20:56:13,769 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea, entries=100, sequenceid=232, filesize=9.4 K 2023-03-28 20:56:13,770 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=140.89 KB/144270 for 1f0e2462e279947a10f94052c3152427 in 455ms, sequenceid=232, compaction requested=true 2023-03-28 20:56:13,770 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:13,770 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:13,770 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:13,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=147.60 KB heapSize=418.41 KB 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:13,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-28 20:56:13,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:13,773 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aed9f47[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:13,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:13,776 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,777 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:13,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,777 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:13,778 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,778 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:13,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,779 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:13,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037033773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037033774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037033775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037033774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037033776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:13,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:13,782 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:13,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:13,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:13,783 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:13,783 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:13,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,787 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,790 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:13,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:13,795 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,795 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:13,795 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:13,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037033880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037033880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037033882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037033885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:13,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037033885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,895 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:13,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:13,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:13,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:13,899 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,901 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,901 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:13,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:13,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:13,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:13,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:13,902 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:13,903 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,903 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:13,903 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:13,903 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:13,909 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,909 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:13,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d93e024[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:13,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:13,911 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,912 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,913 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:13,914 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:13,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:13,917 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:13,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:13,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:13,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:13,917 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:13,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,918 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:13,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:13,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,921 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:13,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:13,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:13,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:13,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:13,924 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:13,924 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:13,924 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:13,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,024 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,028 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,028 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,029 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,030 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,030 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,031 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,031 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,033 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,033 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398c2f44[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,035 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,036 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,037 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,037 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,037 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,039 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,039 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,039 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,039 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,041 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,041 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,042 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,042 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,043 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,044 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,048 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,048 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,048 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,049 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,049 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,050 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,050 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037034085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037034086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037034088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037034090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037034090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,150 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,153 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,153 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,153 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,153 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,154 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,154 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,155 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,155 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,156 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,157 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,158 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,158 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,158 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,158 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,158 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,161 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,161 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507393c4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,164 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,166 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,166 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,167 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,167 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,170 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,171 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,171 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,172 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,173 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,174 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,179 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,179 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,180 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,180 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,180 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,195 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303286029ec8d74f34ed3b2388ddca1f6a04d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286029ec8d74f34ed3b2388ddca1f6a04d 2023-03-28 20:56:14,201 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/9efca908c5e049f68bb1d1f67cd8b205 2023-03-28 20:56:14,280 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,284 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,285 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,285 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,286 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,287 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,293 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,293 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,293 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,293 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,293 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,295 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632292a[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,297 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,298 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,298 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,300 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,300 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,302 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,302 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,303 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,303 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,304 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,305 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,307 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,310 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,309 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,310 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,310 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,310 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037034388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037034392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037034392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037034394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037034396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,410 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,413 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,413 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,414 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,415 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,415 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,417 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,420 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a27b08[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,421 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,423 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,424 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,424 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,425 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,426 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,427 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,428 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,430 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,431 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,431 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,431 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,431 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,531 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,535 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,535 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,536 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,537 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,539 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,541 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0c82d7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,542 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,543 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,544 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,547 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,549 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,549 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,550 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,550 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,552 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,553 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,553 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,555 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,557 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,557 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,558 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,558 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,558 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/5df85744151a41bc9342e74b578e36c8 2023-03-28 20:56:14,658 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,661 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,661 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,662 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,667 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,667 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,668 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,669 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,671 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102dd721[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,673 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,674 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,674 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,674 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,674 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,674 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,674 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,676 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,678 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,678 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,678 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,678 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,678 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,680 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,680 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,680 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,681 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,682 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,684 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,684 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,685 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,685 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,685 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,685 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,785 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,788 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,788 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,789 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,791 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,791 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,793 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,793 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,793 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,793 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,793 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,795 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,795 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,795 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,795 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1c777[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,797 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,799 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,799 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,801 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,804 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,804 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,804 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,807 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,808 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,809 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,813 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,813 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,813 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,814 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037034895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037034898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037034899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037034900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037034900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,914 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:14,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:14,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:14,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:14,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:14,917 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:14,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:14,917 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:14,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,918 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,920 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,920 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:14,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:14,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:14,921 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:14,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:14,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:14,922 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:14,922 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:14,924 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac86bc[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:14,926 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:14,926 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:14,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,928 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:14,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:14,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,929 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,929 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:14,932 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:14,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:14,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:14,932 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:14,932 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:14,934 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,934 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,935 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:14,935 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,937 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:14,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:14,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:14,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:14,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:14,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:14,940 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:14,940 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:14,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:14,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,941 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:14,941 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:14,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:14,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:14,941 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/3be552f2b5a24cdbad3aa5d785c7cebb 2023-03-28 20:56:15,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/9efca908c5e049f68bb1d1f67cd8b205 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205 2023-03-28 20:56:15,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205, entries=250, sequenceid=510, filesize=39.2 K 2023-03-28 20:56:15,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/5df85744151a41bc9342e74b578e36c8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8 2023-03-28 20:56:15,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8, entries=150, sequenceid=510, filesize=11.9 K 2023-03-28 20:56:15,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/3be552f2b5a24cdbad3aa5d785c7cebb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb 2023-03-28 20:56:15,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb, entries=150, sequenceid=510, filesize=11.9 K 2023-03-28 20:56:15,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~418.36 KB/428400, currentSize=33.54 KB/34350 for 1f0e2462e279947a10f94052c3152427 in 1267ms, sequenceid=510, compaction requested=true 2023-03-28 20:56:15,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:15,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:15,038 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-28 20:56:15,038 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-28 20:56:15,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:15,041 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 195878 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-28 20:56:15,041 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82384 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-28 20:56:15,041 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:15,041 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:15,041 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:15,041 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,041 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,041 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5952d8f0e433486992fc92822aea7a4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=80.5 K 2023-03-28 20:56:15,041 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e1be5c961e1844afba8a2768145c63e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=191.3 K 2023-03-28 20:56:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:15,042 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5952d8f0e433486992fc92822aea7a4a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:15,042 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e1be5c961e1844afba8a2768145c63e9, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:15,042 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 7268b624ce384033884bbe9d5dcad31a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680036970304 2023-03-28 20:56:15,042 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting bae6277df83b4980b7f42f3ec0ef9269, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680036970304 2023-03-28 20:56:15,043 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e63d14e21ae847da90a19aaa071aec1c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680036970426 2023-03-28 20:56:15,043 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4043c8451a884496b439dac9ba5e992d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680036970426 2023-03-28 20:56:15,043 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 28d171aea4ea4996ba1bba7f4cb9725b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680036971555 2023-03-28 20:56:15,043 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 33cb7d88de944b549b28db32cc694613, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680036971555 2023-03-28 20:56:15,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:15,044 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 95790a13384b41b4b710f6e3ff9f8cf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680036972178 2023-03-28 20:56:15,044 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e68d9bcac60a4c26a578f291d812b372, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680036972178 2023-03-28 20:56:15,044 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f0c684b71c2b4d449cdf842164b539dc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680036973313 2023-03-28 20:56:15,044 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 7e5076b9c3514eec8fa9dddf84b22157, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680036973312 2023-03-28 20:56:15,044 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5df85744151a41bc9342e74b578e36c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:15,045 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 9efca908c5e049f68bb1d1f67cd8b205, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:15,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:15,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:15,045 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:15,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,046 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,047 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,047 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:15,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:15,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:15,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:15,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:15,049 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:15,050 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:15,050 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,069 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#611 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:15,079 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328f294b2436484478788eb79cf1c1d5956 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f294b2436484478788eb79cf1c1d5956 2023-03-28 20:56:15,082 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#610 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:15,099 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/21598a97fcc147eaae64397108239a3a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/21598a97fcc147eaae64397108239a3a 2023-03-28 20:56:15,103 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into 21598a97fcc147eaae64397108239a3a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:15,103 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,103 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=9, startTime=1680036975038; duration=0sec 2023-03-28 20:56:15,103 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:15,103 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-28 20:56:15,105 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82384 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-28 20:56:15,105 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/58baa96990d4437980a3ccf765f6b382 2023-03-28 20:56:15,105 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:15,106 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,106 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/66c914b3a8fb4bf5aa5f78b3cb842960, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=80.5 K 2023-03-28 20:56:15,106 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 66c914b3a8fb4bf5aa5f78b3cb842960, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680036969691 2023-03-28 20:56:15,107 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1763992fe52e4155b144eafbefe17887, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680036970304 2023-03-28 20:56:15,107 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 92193ad7c9e644cea5e280452df5be40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680036970426 2023-03-28 20:56:15,108 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting cd929509cbe24fbb8d9dc467a5bb9bce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680036971555 2023-03-28 20:56:15,109 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e5be5ac46dca4e82ac448bb85f99b658, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680036972178 2023-03-28 20:56:15,109 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 80bc338d20cf4c9491dd9689f99580ea, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680036973313 2023-03-28 20:56:15,110 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3be552f2b5a24cdbad3aa5d785c7cebb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:15,135 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#613 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:15,135 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/b1eebae9d402443ab093a630eba73daa as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b1eebae9d402443ab093a630eba73daa 2023-03-28 20:56:15,137 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/b32c60cbf3af41688506133cd97f14e1 2023-03-28 20:56:15,145 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into b1eebae9d402443ab093a630eba73daa(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:15,145 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,145 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=9, startTime=1680036975038; duration=0sec 2023-03-28 20:56:15,145 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:15,150 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc 2023-03-28 20:56:15,154 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into ae8ac9ddb2cb40ae8de02e7c03c44cdc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:15,154 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,154 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=9, startTime=1680036975038; duration=0sec 2023-03-28 20:56:15,155 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:15,554 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/5c5ca5668e7341dc81a42b7ef9d7583c 2023-03-28 20:56:15,558 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/58baa96990d4437980a3ccf765f6b382 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382 2023-03-28 20:56:15,562 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382, entries=150, sequenceid=518, filesize=25.4 K 2023-03-28 20:56:15,562 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/b32c60cbf3af41688506133cd97f14e1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1 2023-03-28 20:56:15,565 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1, entries=150, sequenceid=518, filesize=11.9 K 2023-03-28 20:56:15,566 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/5c5ca5668e7341dc81a42b7ef9d7583c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c 2023-03-28 20:56:15,573 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c, entries=150, sequenceid=518, filesize=11.9 K 2023-03-28 20:56:15,574 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 1f0e2462e279947a10f94052c3152427 in 524ms, sequenceid=518, compaction requested=false 2023-03-28 20:56:15,574 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,574 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:15,574 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:15,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:15,575 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:15,576 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,576 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:15,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288ea444[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:15,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:15,578 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,582 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:15,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:15,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,583 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:15,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:15,584 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:15,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:15,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:15,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:15,587 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:15,587 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:15,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,588 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,589 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,590 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,592 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:15,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:15,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:15,592 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,593 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:15,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,593 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:15,593 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,693 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:15,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:15,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:15,697 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:15,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:15,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:15,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,697 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,699 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,699 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,700 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:15,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:15,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:15,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,701 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:15,702 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,702 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:15,703 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,703 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:15,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e4eaab[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:15,705 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:15,705 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,706 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:15,707 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,707 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:15,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:15,708 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,708 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:15,711 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:15,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:15,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:15,711 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:15,711 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:15,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,713 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,714 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,715 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:15,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:15,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,718 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:15,718 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:15,718 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,818 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:15,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:15,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:15,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:15,822 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:15,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:15,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:15,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,822 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,824 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,824 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:15,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:15,826 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:15,826 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,826 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:15,827 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:15,827 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:15,829 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,829 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f17757f[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:15,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:15,831 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,833 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:15,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,833 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,835 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:15,837 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:15,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:15,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:15,838 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:15,838 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:15,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,839 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,841 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,842 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:15,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,844 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,844 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,845 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:15,845 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,845 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,845 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:15,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:15,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:15,943 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328174660d23ff54a8495550ce51e9cde73 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328174660d23ff54a8495550ce51e9cde73 2023-03-28 20:56:15,945 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:15,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:15,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:15,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:15,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:15,949 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:15,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:15,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:15,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,949 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:15,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037035948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036975944, max timestamp=1680036975951Num uniques 859; 2023-03-28 20:56:15,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:56:15,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680036975944, max timestamp=1680036975951Num uniques 859; 2023-03-28 20:56:15,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-28 20:56:15,953 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680036975945, max timestamp=1680036975951Num uniques 859; 2023-03-28 20:56:15,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-28 20:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:15,954 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,955 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:15,960 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:15,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:15,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:15,964 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=534, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/d523db1102744fff8b818bde297b3d0c 2023-03-28 20:56:15,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:15,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:15,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:15,964 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:15,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:15,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:15,965 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:15,967 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfa9147[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:15,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:15,969 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:15,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:15,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:15,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037035967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037035968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:15,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037035971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:15,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037035973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0b57555dc2de48e19417952c1a5a9e91 2023-03-28 20:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:15,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:15,983 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:15,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:15,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:15,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:15,983 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:15,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:15,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:15,987 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:15,991 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:15,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:15,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:15,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:15,993 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:15,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:15,994 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:15,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:15,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:15,994 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:15,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037036053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037036073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037036074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037036076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037036076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,094 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:16,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:16,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:16,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:16,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:16,097 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:16,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:16,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:16,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,097 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,098 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:16,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:16,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:16,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:16,112 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:16,114 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,114 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760bacf8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:16,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:16,116 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,117 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:16,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:16,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:16,119 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,119 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:16,121 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:16,121 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:16,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:16,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:16,122 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:16,123 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,123 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,123 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,124 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,126 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,129 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:16,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,128 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,129 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,129 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:16,129 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:16,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:16,229 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:16,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:16,233 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:16,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:16,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:16,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,234 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:16,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:16,237 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,237 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:16,238 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:16,238 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:16,239 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,239 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bb120[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:16,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:16,242 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,243 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:16,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:16,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:16,243 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:16,245 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,245 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:16,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,246 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:16,248 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:16,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:16,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:16,248 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:16,249 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:16,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,252 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,253 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,255 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,255 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,256 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:16,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:16,256 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:16,256 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:16,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037036257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037036277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037036278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037036280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037036280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,356 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:16,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:16,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:16,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:16,360 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:16,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:16,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:16,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,361 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,361 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,362 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:16,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:16,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:16,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:16,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:16,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:16,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:16,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:16,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:16,367 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6393803b[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:16,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:16,369 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,370 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:16,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:16,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,372 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:16,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:16,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:16,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:16,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:16,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:16,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:16,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,377 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,379 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:16,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:16,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,390 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,391 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:16,391 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:16,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,390 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f6b8eba24de848e5bbadb34e3f210971 2023-03-28 20:56:16,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/d523db1102744fff8b818bde297b3d0c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c 2023-03-28 20:56:16,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c, entries=400, sequenceid=534, filesize=59.8 K 2023-03-28 20:56:16,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0b57555dc2de48e19417952c1a5a9e91 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91 2023-03-28 20:56:16,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91, entries=150, sequenceid=534, filesize=11.9 K 2023-03-28 20:56:16,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f6b8eba24de848e5bbadb34e3f210971 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971 2023-03-28 20:56:16,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971, entries=150, sequenceid=534, filesize=11.9 K 2023-03-28 20:56:16,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.31 KB/132415 for 1f0e2462e279947a10f94052c3152427 in 491ms, sequenceid=534, compaction requested=true 2023-03-28 20:56:16,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:16,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:16,420 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:16,420 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:16,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:16,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:16,421 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113960 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:16,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:16,421 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:16,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:16,422 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,422 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,422 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b1eebae9d402443ab093a630eba73daa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=111.3 K 2023-03-28 20:56:16,422 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/21598a97fcc147eaae64397108239a3a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.3 K 2023-03-28 20:56:16,422 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 21598a97fcc147eaae64397108239a3a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:16,422 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting b1eebae9d402443ab093a630eba73daa, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:16,422 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting b32c60cbf3af41688506133cd97f14e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680036973773 2023-03-28 20:56:16,423 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 58baa96990d4437980a3ccf765f6b382, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680036973773 2023-03-28 20:56:16,423 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0b57555dc2de48e19417952c1a5a9e91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975920 2023-03-28 20:56:16,423 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d523db1102744fff8b818bde297b3d0c, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975917 2023-03-28 20:56:16,430 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#618 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:16,434 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#619 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:16,445 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/296f5a2cfddf4552875963feb62ae7de as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/296f5a2cfddf4552875963feb62ae7de 2023-03-28 20:56:16,448 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into 296f5a2cfddf4552875963feb62ae7de(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:16,448 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:16,448 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=13, startTime=1680036976420; duration=0sec 2023-03-28 20:56:16,448 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:16,448 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:16,449 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:16,449 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:16,449 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,449 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.3 K 2023-03-28 20:56:16,449 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ae8ac9ddb2cb40ae8de02e7c03c44cdc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680036973761 2023-03-28 20:56:16,450 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5c5ca5668e7341dc81a42b7ef9d7583c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680036973773 2023-03-28 20:56:16,450 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f6b8eba24de848e5bbadb34e3f210971, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975920 2023-03-28 20:56:16,455 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#620 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:16,462 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/c70f7fe5cfb542179f0e968eedb08063 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c70f7fe5cfb542179f0e968eedb08063 2023-03-28 20:56:16,467 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into c70f7fe5cfb542179f0e968eedb08063(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:16,467 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:16,467 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=13, startTime=1680036976420; duration=0sec 2023-03-28 20:56:16,467 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:16,491 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:16,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:16,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:16,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:16,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:16,494 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:16,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:16,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:16,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,496 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,496 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:16,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:16,497 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,497 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:16,498 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=129.31 KB heapSize=364.54 KB 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:16,498 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:16,518 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303285e848e2eb5214f64a43557b4e8adbf73 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e848e2eb5214f64a43557b4e8adbf73 2023-03-28 20:56:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:16,563 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037036581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037036582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037036583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037036583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037036585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037036685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037036686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,856 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/d3851bfaadf44500a5f75ecf84f526ae as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d3851bfaadf44500a5f75ecf84f526ae 2023-03-28 20:56:16,861 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into d3851bfaadf44500a5f75ecf84f526ae(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:16,861 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:16,861 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=13, startTime=1680036976420; duration=0sec 2023-03-28 20:56:16,861 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037036888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:16,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037036889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,934 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=571, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/5e61af4c55d649e193133df9ffe6afba 2023-03-28 20:56:16,945 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/2956e8a771444a19838d4886b84c6112 2023-03-28 20:56:16,955 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/12f3de1272e74979b9ea188cd493965f 2023-03-28 20:56:16,959 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/5e61af4c55d649e193133df9ffe6afba as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba 2023-03-28 20:56:16,963 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba, entries=150, sequenceid=571, filesize=25.4 K 2023-03-28 20:56:16,964 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/2956e8a771444a19838d4886b84c6112 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112 2023-03-28 20:56:16,967 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112, entries=150, sequenceid=571, filesize=11.9 K 2023-03-28 20:56:16,968 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/12f3de1272e74979b9ea188cd493965f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f 2023-03-28 20:56:16,971 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f, entries=150, sequenceid=571, filesize=11.9 K 2023-03-28 20:56:16,972 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.31 KB/132415, heapSize ~364.49 KB/373240, currentSize=53.67 KB/54960 for 1f0e2462e279947a10f94052c3152427 in 475ms, sequenceid=571, compaction requested=false 2023-03-28 20:56:16,972 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:16,972 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:16,974 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9c55a7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:16,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:16,976 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,978 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:16,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,978 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:16,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:16,980 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:16,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:16,985 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:16,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:16,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:16,985 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:16,985 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:16,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,987 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:16,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,988 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:16,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,989 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:16,990 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,993 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:16,994 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:16,994 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:16,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:16,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:16,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:16,994 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:16,995 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:17,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,095 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:17,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:17,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:17,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:17,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:17,099 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:17,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:17,099 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:17,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,101 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,102 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:17,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:17,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:17,104 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,105 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:17,105 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:17,105 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:17,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@375b56a0[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:17,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:17,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,111 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:17,111 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:17,111 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,112 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:17,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037037109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037037110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:17,115 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:17,115 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:17,115 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:17,115 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:17,115 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:17,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037037113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,117 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,117 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,117 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,118 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,119 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:17,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,122 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,122 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,123 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,123 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:17,123 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:17,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:17,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037037191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037037193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037037213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037037213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037037218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,223 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:17,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:17,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:17,226 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:17,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:17,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:17,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,227 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,228 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,228 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:17,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:17,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:17,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:17,230 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,230 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:17,230 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:17,230 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:17,231 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc333da[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:17,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:17,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,234 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,235 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:17,235 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:17,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,235 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:17,235 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:17,237 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,237 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:17,241 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:17,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:17,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:17,241 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:17,241 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:17,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,242 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,243 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,244 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,250 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:17,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,250 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,250 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:17,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,250 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:17,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,350 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:17,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:17,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:17,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:17,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:17,354 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:17,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:17,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:17,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,354 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,356 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:17,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:17,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:17,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:17,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:17,364 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:17,366 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,366 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,367 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:17,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12aee401[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:17,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:17,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,369 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:17,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:17,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:17,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:17,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:17,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:17,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:17,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:17,374 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:17,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:17,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,376 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,378 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,379 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,381 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,382 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:17,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:17,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:17,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,382 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:17,382 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:17,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037037417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037037418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037037421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,482 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:17,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:17,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:17,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:17,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:17,485 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:17,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:17,485 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:17,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,486 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,487 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,487 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:17,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:17,489 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,489 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:17,489 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:17,489 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:17,490 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,491 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c61ba9e[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:17,492 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:17,492 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,494 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:17,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:17,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:17,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,494 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:17,495 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:17,498 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:17,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:17,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:17,498 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:17,498 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:17,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,499 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,500 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:17,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:17,500 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:17,502 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032826c16fbf3e41432191d74f9cdeea2c21 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032826c16fbf3e41432191d74f9cdeea2c21 2023-03-28 20:56:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:17,502 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:17,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,504 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:17,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,505 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:17,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:17,505 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,505 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:17,505 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:17,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:17,507 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=585, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/84e542a85a23408496896f5a3fdcc803 2023-03-28 20:56:17,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0c6ae91edb5a422984eed980c3ed998b 2023-03-28 20:56:17,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/5eeed48d9c35460db5282215a9feaaeb 2023-03-28 20:56:17,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/84e542a85a23408496896f5a3fdcc803 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803 2023-03-28 20:56:17,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803, entries=150, sequenceid=585, filesize=25.4 K 2023-03-28 20:56:17,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0c6ae91edb5a422984eed980c3ed998b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b 2023-03-28 20:56:17,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b, entries=150, sequenceid=585, filesize=11.9 K 2023-03-28 20:56:17,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/5eeed48d9c35460db5282215a9feaaeb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb 2023-03-28 20:56:17,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb, entries=150, sequenceid=585, filesize=11.9 K 2023-03-28 20:56:17,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1f0e2462e279947a10f94052c3152427 in 446ms, sequenceid=585, compaction requested=true 2023-03-28 20:56:17,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:17,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:17,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:17,536 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:17,536 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:17,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:17,538 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:17,538 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:17,538 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,538 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d3851bfaadf44500a5f75ecf84f526ae, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=77.0 K 2023-03-28 20:56:17,538 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:17,538 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:17,538 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,538 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/296f5a2cfddf4552875963feb62ae7de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.4 K 2023-03-28 20:56:17,538 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting d3851bfaadf44500a5f75ecf84f526ae, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975920 2023-03-28 20:56:17,539 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 296f5a2cfddf4552875963feb62ae7de, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975920 2023-03-28 20:56:17,539 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 5e61af4c55d649e193133df9ffe6afba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680036975959 2023-03-28 20:56:17,539 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2956e8a771444a19838d4886b84c6112, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680036975959 2023-03-28 20:56:17,539 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 84e542a85a23408496896f5a3fdcc803, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:17,539 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0c6ae91edb5a422984eed980c3ed998b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:17,547 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#628 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:17,561 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#627 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:17,580 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/fd2d8e7cd5e14af48c1f65923f9e47dc as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/fd2d8e7cd5e14af48c1f65923f9e47dc 2023-03-28 20:56:17,584 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into fd2d8e7cd5e14af48c1f65923f9e47dc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:17,584 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:17,584 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=13, startTime=1680036977536; duration=0sec 2023-03-28 20:56:17,584 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:17,584 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:17,585 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:17,585 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:17,585 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,585 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c70f7fe5cfb542179f0e968eedb08063, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.4 K 2023-03-28 20:56:17,586 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting c70f7fe5cfb542179f0e968eedb08063, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680036975920 2023-03-28 20:56:17,586 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 12f3de1272e74979b9ea188cd493965f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680036975959 2023-03-28 20:56:17,587 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 5eeed48d9c35460db5282215a9feaaeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:17,603 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:17,605 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:17,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:17,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:17,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:17,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:17,612 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:17,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:17,612 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:17,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,613 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,615 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:17,615 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:17,615 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,615 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:17,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:17,619 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/688d1906d8a54d288c32ae9c87f0f957 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/688d1906d8a54d288c32ae9c87f0f957 2023-03-28 20:56:17,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:17,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:17,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:17,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:17,620 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:17,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:17,621 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:17,621 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:17,621 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:17,627 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/72f84f295f6140929e57a52b3aa95d31 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/72f84f295f6140929e57a52b3aa95d31 2023-03-28 20:56:17,627 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into 688d1906d8a54d288c32ae9c87f0f957(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:17,627 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:17,628 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=13, startTime=1680036977536; duration=0sec 2023-03-28 20:56:17,628 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:17,632 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into 72f84f295f6140929e57a52b3aa95d31(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:17,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:17,632 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=13, startTime=1680036977536; duration=0sec 2023-03-28 20:56:17,632 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:17,636 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032859eb3157377f418aaabaa64f7f7ee89a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032859eb3157377f418aaabaa64f7f7ee89a 2023-03-28 20:56:17,641 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=605, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e15a808954984fa893f06020c9d7a4e4 2023-03-28 20:56:17,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:17,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:17,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037037709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037037709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037037719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037037720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037037724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037037812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:17,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:17,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037037813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037038015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037038016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,054 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/331276eb3393463cad7e15227318bb96 2023-03-28 20:56:18,063 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f0c84b9bd9ec49109d407e7611fc0679 2023-03-28 20:56:18,067 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/e15a808954984fa893f06020c9d7a4e4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4 2023-03-28 20:56:18,070 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4, entries=150, sequenceid=605, filesize=25.4 K 2023-03-28 20:56:18,071 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/331276eb3393463cad7e15227318bb96 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96 2023-03-28 20:56:18,074 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96, entries=150, sequenceid=605, filesize=11.9 K 2023-03-28 20:56:18,074 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f0c84b9bd9ec49109d407e7611fc0679 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679 2023-03-28 20:56:18,077 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679, entries=150, sequenceid=605, filesize=11.9 K 2023-03-28 20:56:18,078 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1f0e2462e279947a10f94052c3152427 in 457ms, sequenceid=605, compaction requested=false 2023-03-28 20:56:18,078 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,078 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:18,080 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e21fe4[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:18,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:18,082 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,083 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:18,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,083 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:18,084 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:18,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:18,085 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:18,087 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:18,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:18,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:18,087 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:18,087 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:18,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,090 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:18,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,091 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:18,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:18,094 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:18,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,094 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,094 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:18,094 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:18,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:18,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:18,194 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:18,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:18,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:18,198 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:18,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:18,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:18,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:18,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,198 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,202 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:18,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:18,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,202 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:18,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:18,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:18,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:18,203 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:18,204 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:18,204 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,217 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328cb48348fc6da4a07af4e7943959873e4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb48348fc6da4a07af4e7943959873e4 2023-03-28 20:56:18,221 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=620, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/28ecdd5330264f5187d282f542aff5af 2023-03-28 20:56:18,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:18,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:18,231 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/d487cbae9ac64ff28c25c4f0e874184c 2023-03-28 20:56:18,249 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/a3a0080f4ee84034bb9e7d5d56abc348 2023-03-28 20:56:18,253 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/28ecdd5330264f5187d282f542aff5af as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af 2023-03-28 20:56:18,256 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af, entries=150, sequenceid=620, filesize=25.4 K 2023-03-28 20:56:18,257 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/d487cbae9ac64ff28c25c4f0e874184c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c 2023-03-28 20:56:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037038255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037038256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037038256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,260 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c, entries=150, sequenceid=620, filesize=11.9 K 2023-03-28 20:56:18,261 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/a3a0080f4ee84034bb9e7d5d56abc348 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348 2023-03-28 20:56:18,265 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348, entries=150, sequenceid=620, filesize=11.9 K 2023-03-28 20:56:18,265 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 1f0e2462e279947a10f94052c3152427 in 61ms, sequenceid=620, compaction requested=true 2023-03-28 20:56:18,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,265 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:18,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:18,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:18,265 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,270 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:18,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754bb1e1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:18,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:18,272 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,273 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:18,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,273 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:18,274 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:18,277 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:18,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:18,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:18,277 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:18,277 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:18,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,278 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:18,279 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,280 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,283 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:18,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,284 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:18,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:18,284 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:18,284 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,284 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:18,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:18,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:18,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:18,331 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328239c8fa7c8bd486fa1ef0861fb04606a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328239c8fa7c8bd486fa1ef0861fb04606a 2023-03-28 20:56:18,336 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=642, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/146370775ecf411794cd09e360851f4f 2023-03-28 20:56:18,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037038337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037038337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/2326380e2756453d9df0658371c914e5 2023-03-28 20:56:18,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/ad9e836a633b43aa839efd73b82ad76c 2023-03-28 20:56:18,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/146370775ecf411794cd09e360851f4f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f 2023-03-28 20:56:18,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037038359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037038359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037038359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f, entries=200, sequenceid=642, filesize=32.3 K 2023-03-28 20:56:18,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/2326380e2756453d9df0658371c914e5 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5 2023-03-28 20:56:18,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5, entries=150, sequenceid=642, filesize=11.9 K 2023-03-28 20:56:18,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/ad9e836a633b43aa839efd73b82ad76c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c 2023-03-28 20:56:18,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c, entries=150, sequenceid=642, filesize=11.9 K 2023-03-28 20:56:18,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for 1f0e2462e279947a10f94052c3152427 in 50ms, sequenceid=642, compaction requested=true 2023-03-28 20:56:18,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:18,370 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:18,370 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:18,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:18,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:18,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112042 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:18,371 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:18,371 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,372 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/72f84f295f6140929e57a52b3aa95d31, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=109.4 K 2023-03-28 20:56:18,372 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:18,372 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 72f84f295f6140929e57a52b3aa95d31, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:18,372 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:18,372 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,372 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/fd2d8e7cd5e14af48c1f65923f9e47dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=48.4 K 2023-03-28 20:56:18,372 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting e15a808954984fa893f06020c9d7a4e4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680036977102 2023-03-28 20:56:18,373 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting fd2d8e7cd5e14af48c1f65923f9e47dc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:18,373 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 28ecdd5330264f5187d282f542aff5af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680036977706 2023-03-28 20:56:18,373 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 331276eb3393463cad7e15227318bb96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680036977102 2023-03-28 20:56:18,373 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 146370775ecf411794cd09e360851f4f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:18,373 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d487cbae9ac64ff28c25c4f0e874184c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680036977706 2023-03-28 20:56:18,374 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 2326380e2756453d9df0658371c914e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:18,382 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#640 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:18,383 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#639 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:18,384 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:18,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:18,388 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:18,388 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:18,388 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:18,388 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:18,390 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,390 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,391 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,391 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:18,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:18,392 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,392 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:18,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:18,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:18,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:18,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:18,396 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:18,400 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,400 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:18,400 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-28 20:56:18,400 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:18,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:18,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:18,401 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:18,414 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/ac5594fdffc3400686561b7dd83ae5e8 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/ac5594fdffc3400686561b7dd83ae5e8 2023-03-28 20:56:18,418 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into ac5594fdffc3400686561b7dd83ae5e8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:18,419 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,419 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=12, startTime=1680036978370; duration=0sec 2023-03-28 20:56:18,419 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:18,419 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:18,420 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:18,420 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:18,420 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:18,420 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/688d1906d8a54d288c32ae9c87f0f957, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=48.4 K 2023-03-28 20:56:18,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 688d1906d8a54d288c32ae9c87f0f957, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1680036976571 2023-03-28 20:56:18,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f0c84b9bd9ec49109d407e7611fc0679, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680036977102 2023-03-28 20:56:18,421 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting a3a0080f4ee84034bb9e7d5d56abc348, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680036977706 2023-03-28 20:56:18,422 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ad9e836a633b43aa839efd73b82ad76c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:18,430 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#642 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:18,438 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328bc08bc122e894dd1b8cee77212c9b693 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328bc08bc122e894dd1b8cee77212c9b693 2023-03-28 20:56:18,439 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/605eec9d048344fc94f4c38299124d95 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/605eec9d048344fc94f4c38299124d95 2023-03-28 20:56:18,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:18,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:18,443 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into 605eec9d048344fc94f4c38299124d95(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:18,443 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,443 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=12, startTime=1680036978370; duration=0sec 2023-03-28 20:56:18,444 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:18,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036978458, max timestamp=1680036978466Num uniques 859; 2023-03-28 20:56:18,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036978458, max timestamp=1680036978466Num uniques 859; 2023-03-28 20:56:18,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680036978458, max timestamp=1680036978466Num uniques 859; 2023-03-28 20:56:18,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:56:18,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:56:18,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-28 20:56:18,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037038492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037038493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037038562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037038562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037038562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037038594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037038596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037038797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037038800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,833 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/4c600025df494d63886ebfbdc1b3dcd4 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c600025df494d63886ebfbdc1b3dcd4 2023-03-28 20:56:18,836 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into 4c600025df494d63886ebfbdc1b3dcd4(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:18,836 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:18,836 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=12, startTime=1680036978370; duration=0sec 2023-03-28 20:56:18,837 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:18,845 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=652, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1677833a3f004a41998c4783d123e65f 2023-03-28 20:56:18,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037038865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037038866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:18,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037038867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037039100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037039103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,253 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/1862a3ed2d8f493498c87aac9c1e1206 2023-03-28 20:56:19,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037039368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037039371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037039372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037039604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037039607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,662 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/2b8daf9133c244bf97d5c2768ffa378c 2023-03-28 20:56:19,667 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/1677833a3f004a41998c4783d123e65f as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f 2023-03-28 20:56:19,670 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f, entries=150, sequenceid=652, filesize=25.4 K 2023-03-28 20:56:19,671 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/1862a3ed2d8f493498c87aac9c1e1206 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206 2023-03-28 20:56:19,674 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206, entries=150, sequenceid=652, filesize=11.9 K 2023-03-28 20:56:19,675 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/2b8daf9133c244bf97d5c2768ffa378c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c 2023-03-28 20:56:19,678 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c, entries=150, sequenceid=652, filesize=11.9 K 2023-03-28 20:56:19,678 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1f0e2462e279947a10f94052c3152427 in 1278ms, sequenceid=652, compaction requested=false 2023-03-28 20:56:19,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:19,679 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,680 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a0dda5[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:19,682 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:19,682 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,685 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:19,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,685 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:19,686 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:19,689 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:19,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:19,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:19,689 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:19,689 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:19,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:19,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,690 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:19,691 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,693 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:19,695 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:19,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:19,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:19,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:19,695 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:19,696 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:19,696 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:19,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:19,796 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:19,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:19,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:19,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:19,799 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:19,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:19,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:19,799 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:19,800 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,800 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:19,802 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,802 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:19,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:19,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:19,803 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:19,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:19,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:19,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:19,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:19,805 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:19,805 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-28 20:56:20,219 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303285d5936f9e4514a4fac1663eab7614bfa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285d5936f9e4514a4fac1663eab7614bfa 2023-03-28 20:56:20,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:20,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:20,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037040385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037040386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037040387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037040487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037040489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037040490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037040613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037040614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,624 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=691, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/44eb69e5a4b4477b833f0ffef5230161 2023-03-28 20:56:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037040691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037040692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037040693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037040996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037040996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037040996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,033 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0d7678fbf3c84e328f0fc04dffbab4bb 2023-03-28 20:56:21,443 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/52cb47f6274948238fef6736ed648344 2023-03-28 20:56:21,447 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/44eb69e5a4b4477b833f0ffef5230161 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161 2023-03-28 20:56:21,451 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161, entries=150, sequenceid=691, filesize=25.4 K 2023-03-28 20:56:21,451 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/0d7678fbf3c84e328f0fc04dffbab4bb as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb 2023-03-28 20:56:21,454 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb, entries=150, sequenceid=691, filesize=11.9 K 2023-03-28 20:56:21,455 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/52cb47f6274948238fef6736ed648344 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344 2023-03-28 20:56:21,458 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344, entries=150, sequenceid=691, filesize=11.9 K 2023-03-28 20:56:21,459 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1f0e2462e279947a10f94052c3152427 in 1654ms, sequenceid=691, compaction requested=true 2023-03-28 20:56:21,459 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:21,459 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:21,461 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f42065[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:21,463 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:21,463 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,464 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:21,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,464 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:21,466 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:21,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,466 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:21,469 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:21,469 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:21,469 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:21,469 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:21,469 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:21,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,471 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,472 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,474 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:21,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:21,477 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,477 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:21,477 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,478 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,477 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:21,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:21,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037041525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037041527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037041528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,578 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:21,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:21,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:21,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:21,581 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:21,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:21,581 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:21,582 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,582 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,583 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,583 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,584 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:21,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:21,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:21,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:21,585 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:21,596 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60781c80[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:21,598 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:21,598 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,599 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:21,599 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,599 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:21,600 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:21,603 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:21,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:21,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:21,603 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:21,603 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:21,605 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,605 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,605 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,606 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,608 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:21,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:21,611 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,612 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,612 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,611 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,612 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:21,612 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:21,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037041630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037041633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037041633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,712 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:21,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:21,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:21,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:21,715 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:21,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:21,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:21,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,716 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,717 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,717 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,718 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:21,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:21,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:21,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:21,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:21,719 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:21,721 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48946dd7[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:21,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:21,723 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,724 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:21,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:21,724 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:21,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,724 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:21,727 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:21,729 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:21,729 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:21,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:21,730 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:21,730 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:21,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,731 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,732 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,734 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,736 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:21,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,736 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:21,737 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,737 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,737 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,736 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:21,837 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:21,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037041835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:21,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:21,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037041837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:21,840 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:21,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:21,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:21,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037041837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,840 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,842 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,842 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,842 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,842 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:21,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:21,843 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,843 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:21,843 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:21,843 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,846 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecfdc95[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:21,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:21,848 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,849 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:21,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,849 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:21,851 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:21,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:21,853 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:21,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:21,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:21,853 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:21,853 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:21,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,854 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,855 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,856 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,859 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,859 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:21,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,860 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:21,860 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:21,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,917 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230328336f826ab43e412da370ff9bade360d1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328336f826ab43e412da370ff9bade360d1 2023-03-28 20:56:21,960 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:21,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:21,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:21,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:21,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:21,963 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:21,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:21,963 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:21,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,964 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,965 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,965 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:21,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:21,967 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,967 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:21,967 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:21,967 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:21,968 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e682821[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:21,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:21,970 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,972 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:21,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,972 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,977 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,977 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:21,979 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:21,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:21,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:21,980 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:21,980 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:21,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,981 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:21,982 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,983 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:21,986 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:21,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:21,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:21,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:21,986 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:21,986 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:21,986 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,086 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:22,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:22,090 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:22,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:22,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:22,090 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,091 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,093 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,093 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:22,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:22,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:22,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:22,094 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,095 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:22,095 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:22,095 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:22,096 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,096 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e54a970[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:22,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:22,098 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,100 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:22,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,100 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:22,101 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:22,104 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:22,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:22,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:22,105 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:22,105 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:22,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,108 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,109 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,110 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:22,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:22,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,112 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,112 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,112 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:22,112 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,113 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037042140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037042141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037042143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,213 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:22,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:22,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:22,216 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:22,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:22,216 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:22,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,217 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,218 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,218 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,218 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,218 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:22,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:22,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:22,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:22,220 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:22,221 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,221 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454593b8[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:22,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:22,224 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,226 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:22,228 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:22,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:22,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,229 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:22,230 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:22,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:22,233 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:22,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:22,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:22,233 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:22,233 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:22,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,235 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,236 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,238 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-28 20:56:22,240 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:22,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:22,240 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,241 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,241 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:22,241 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,323 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=703, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/aa5fd9eb47e548f08521329b4521b9a6 2023-03-28 20:56:22,341 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:22,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:22,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:22,356 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:22,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:22,356 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:22,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/e3bae6f2f20341a7b2234b2c1157b6e1 2023-03-28 20:56:22,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,357 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,359 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,360 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:22,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:22,362 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:22,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:22,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:22,363 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:22,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:22,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:22,363 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:22,365 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/4ecce0b2991c45b0ab22b998890d24c9 2023-03-28 20:56:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654ec4f1[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:22,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:22,368 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,369 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,370 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:22,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/aa5fd9eb47e548f08521329b4521b9a6 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6 2023-03-28 20:56:22,371 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,371 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:22,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6, entries=200, sequenceid=703, filesize=32.3 K 2023-03-28 20:56:22,374 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:22,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:22,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:22,375 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:22,375 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:22,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/e3bae6f2f20341a7b2234b2c1157b6e1 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1 2023-03-28 20:56:22,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1, entries=150, sequenceid=703, filesize=11.9 K 2023-03-28 20:56:22,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/4ecce0b2991c45b0ab22b998890d24c9 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9 2023-03-28 20:56:22,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,381 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:22,382 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9, entries=150, sequenceid=703, filesize=11.9 K 2023-03-28 20:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:22,384 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1f0e2462e279947a10f94052c3152427 in 882ms, sequenceid=703, compaction requested=true 2023-03-28 20:56:22,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:22,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:22,384 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:22,384 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:22,385 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:22,386 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112178 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:22,386 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:22,386 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:22,386 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:22,386 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,386 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,386 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c600025df494d63886ebfbdc1b3dcd4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=109.5 K 2023-03-28 20:56:22,386 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/ac5594fdffc3400686561b7dd83ae5e8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=48.5 K 2023-03-28 20:56:22,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,387 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4c600025df494d63886ebfbdc1b3dcd4, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:22,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:22,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting ac5594fdffc3400686561b7dd83ae5e8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:22,386 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 1677833a3f004a41998c4783d123e65f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680036978327 2023-03-28 20:56:22,387 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:22,387 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 1862a3ed2d8f493498c87aac9c1e1206, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680036978327 2023-03-28 20:56:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:22,387 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 44eb69e5a4b4477b833f0ffef5230161, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680036978464 2023-03-28 20:56:22,387 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 0d7678fbf3c84e328f0fc04dffbab4bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680036978464 2023-03-28 20:56:22,388 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting aa5fd9eb47e548f08521329b4521b9a6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980378 2023-03-28 20:56:22,388 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting e3bae6f2f20341a7b2234b2c1157b6e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980384 2023-03-28 20:56:22,397 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#651 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:22,410 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#652 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:22,438 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/088a500f8f774c6f881462255e2a413a as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/088a500f8f774c6f881462255e2a413a 2023-03-28 20:56:22,442 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into 088a500f8f774c6f881462255e2a413a(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:22,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:22,442 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=12, startTime=1680036982384; duration=0sec 2023-03-28 20:56:22,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:22,442 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-28 20:56:22,444 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-28 20:56:22,444 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:22,444 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,444 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/605eec9d048344fc94f4c38299124d95, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=48.5 K 2023-03-28 20:56:22,444 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 605eec9d048344fc94f4c38299124d95, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1680036978253 2023-03-28 20:56:22,445 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 2b8daf9133c244bf97d5c2768ffa378c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680036978327 2023-03-28 20:56:22,445 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 52cb47f6274948238fef6736ed648344, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680036978464 2023-03-28 20:56:22,445 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 4ecce0b2991c45b0ab22b998890d24c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980384 2023-03-28 20:56:22,452 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#653 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:22,459 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/f8c96febe200401293b4f722a3c55413 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f8c96febe200401293b4f722a3c55413 2023-03-28 20:56:22,463 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into f8c96febe200401293b4f722a3c55413(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:22,463 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:22,463 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=12, startTime=1680036982385; duration=0sec 2023-03-28 20:56:22,463 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:22,487 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:22,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:22,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-28 20:56:22,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-28 20:56:22,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-28 20:56:22,490 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-28 20:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-28 20:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-28 20:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,492 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:22,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:22,493 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:22,493 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,493 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-28 20:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-28 20:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:22,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-28 20:56:22,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-28 20:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-28 20:56:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-28 20:56:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-28 20:56:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-28 20:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. started... 2023-03-28 20:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-28 20:56:22,495 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:22,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:22,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. as already flushing 2023-03-28 20:56:22,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037042637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037042638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037042644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037042645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42693] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037042647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,710 DEBUG [Thread-4145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fdfef3b to 127.0.0.1:56827 2023-03-28 20:56:22,710 DEBUG [Thread-4145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:22,710 DEBUG [Thread-4143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x785e366b to 127.0.0.1:56827 2023-03-28 20:56:22,710 DEBUG [Thread-4143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:22,712 DEBUG [Thread-4141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e4ba6 to 127.0.0.1:56827 2023-03-28 20:56:22,712 DEBUG [Thread-4141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:22,713 DEBUG [Thread-4137] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c1a68cd to 127.0.0.1:56827 2023-03-28 20:56:22,713 DEBUG [Thread-4137] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:22,713 DEBUG [Thread-4139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a709b21 to 127.0.0.1:56827 2023-03-28 20:56:22,713 DEBUG [Thread-4139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:22,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037042741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037042741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,821 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/72a5dc74e7be4965b4bab39ae14aff9b as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/72a5dc74e7be4965b4bab39ae14aff9b 2023-03-28 20:56:22,824 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into 72a5dc74e7be4965b4bab39ae14aff9b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:22,824 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:22,824 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=12, startTime=1680036982384; duration=0sec 2023-03-28 20:56:22,824 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:22,910 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032874904c334b9f4a3cadc774670fda3e45 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032874904c334b9f4a3cadc774670fda3e45 2023-03-28 20:56:22,915 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=725, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/48a9333d271542a885c2a765d4eaca63 2023-03-28 20:56:22,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:22,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037042943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:22,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037042943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:23,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42693] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51940 deadline: 1680037043244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51924 deadline: 1680037043244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,322 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3129baf7cff7454bab4094825628d4e0 2023-03-28 20:56:23,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:23,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51896 deadline: 1680037043650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:23,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51962 deadline: 1680037043651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-28 20:56:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51956 deadline: 1680037043655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,729 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/d3888ce4baeb4ab49ea4faf9a6a628d2 2023-03-28 20:56:23,732 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/48a9333d271542a885c2a765d4eaca63 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63 2023-03-28 20:56:23,735 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63, entries=150, sequenceid=725, filesize=25.4 K 2023-03-28 20:56:23,735 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/3129baf7cff7454bab4094825628d4e0 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0 2023-03-28 20:56:23,739 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0, entries=150, sequenceid=725, filesize=11.9 K 2023-03-28 20:56:23,739 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/d3888ce4baeb4ab49ea4faf9a6a628d2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2 2023-03-28 20:56:23,742 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2, entries=150, sequenceid=725, filesize=11.9 K 2023-03-28 20:56:23,743 INFO [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1f0e2462e279947a10f94052c3152427 in 1248ms, sequenceid=725, compaction requested=false 2023-03-28 20:56:23,743 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:23,743 DEBUG [rs(jenkins-hbase4.apache.org,42693,1680036815956)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-28 20:56:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-28 20:56:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-28 20:56:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-28 20:56:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,746 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-28 20:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42693] regionserver.HRegion(9158): Flush requested on 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:23,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-28 20:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,747 DEBUG [Thread-4134] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:56827 2023-03-28 20:56:23,747 DEBUG [Thread-4134] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:23,748 DEBUG [Thread-4130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a320817 to 127.0.0.1:56827 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:23,748 DEBUG [Thread-4130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:23,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42693,1680036815956' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-28 20:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a28c159[Count = 0] remaining members to acquire global barrier 2023-03-28 20:56:23,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-28 20:56:23,748 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,750 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-28 20:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-28 20:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42693,1680036815956' in zk 2023-03-28 20:56:23,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,750 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-28 20:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-28 20:56:23,753 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,42693,1680036815956' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-28 20:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42693,1680036815956': 2023-03-28 20:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42693,1680036815956' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-28 20:56:23,755 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-28 20:56:23,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-28 20:56:23,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-28 20:56:23,755 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-28 20:56:23,755 INFO [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-28 20:56:23,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-28 20:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-28 20:56:23,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,757 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:23,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-28 20:56:23,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:23,758 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-28 20:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,759 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-28 20:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-28 20:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-28 20:56:23,761 DEBUG [(jenkins-hbase4.apache.org,44521,1680036814691)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-28 20:56:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-28 20:56:23,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-28 20:56:23,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-28 20:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-28 20:56:23,761 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-28 20:56:23,761 DEBUG [Thread-4136] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-28 20:56:23,761 DEBUG [Thread-4136] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-28 20:56:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-28 20:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-28 20:56:23,861 DEBUG [Thread-4136] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-28 20:56:23,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-28 20:56:24,169 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303289a5c0a8705f643edae3e4f6f6bbbaff0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289a5c0a8705f643edae3e4f6f6bbbaff0 2023-03-28 20:56:24,173 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=740, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/885be9a4cfe54a789128059cb192f931 2023-03-28 20:56:24,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/39d5636837f74102b048ab36f868c9da 2023-03-28 20:56:24,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/32480a2d11704652b25141c3cd7d4ed3 2023-03-28 20:56:24,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/885be9a4cfe54a789128059cb192f931 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931 2023-03-28 20:56:24,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931, entries=150, sequenceid=740, filesize=25.4 K 2023-03-28 20:56:24,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/39d5636837f74102b048ab36f868c9da as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da 2023-03-28 20:56:24,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da, entries=150, sequenceid=740, filesize=11.9 K 2023-03-28 20:56:24,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/32480a2d11704652b25141c3cd7d4ed3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3 2023-03-28 20:56:25,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3, entries=150, sequenceid=740, filesize=11.9 K 2023-03-28 20:56:25,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 1f0e2462e279947a10f94052c3152427 in 1253ms, sequenceid=740, compaction requested=true 2023-03-28 20:56:25,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:25,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:25,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:25,001 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:25,001 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:25,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:25,001 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79228 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/A is initiating minor compaction (all files) 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/B is initiating minor compaction (all files) 2023-03-28 20:56:25,002 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/A in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,002 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/B in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,002 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/088a500f8f774c6f881462255e2a413a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=77.4 K 2023-03-28 20:56:25,002 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/72a5dc74e7be4965b4bab39ae14aff9b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.8 K 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 088a500f8f774c6f881462255e2a413a, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980384 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 72a5dc74e7be4965b4bab39ae14aff9b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980384 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 48a9333d271542a885c2a765d4eaca63, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1680036981521 2023-03-28 20:56:25,002 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 3129baf7cff7454bab4094825628d4e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1680036981521 2023-03-28 20:56:25,003 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] compactions.Compactor(207): Compacting 885be9a4cfe54a789128059cb192f931, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680036982634 2023-03-28 20:56:25,003 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 39d5636837f74102b048ab36f868c9da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680036982634 2023-03-28 20:56:25,009 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#B#compaction#661 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:25,010 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#A#compaction#660 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:25,015 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/9222b7c30b34433a84132e519de89b82 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9222b7c30b34433a84132e519de89b82 2023-03-28 20:56:25,019 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/B of 1f0e2462e279947a10f94052c3152427 into 9222b7c30b34433a84132e519de89b82(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:25,019 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:25,019 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/B, priority=13, startTime=1680036985001; duration=0sec 2023-03-28 20:56:25,019 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-28 20:56:25,019 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-28 20:56:25,020 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-28 20:56:25,020 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1912): 1f0e2462e279947a10f94052c3152427/C is initiating minor compaction (all files) 2023-03-28 20:56:25,020 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1f0e2462e279947a10f94052c3152427/C in TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,020 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f8c96febe200401293b4f722a3c55413, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3] into tmpdir=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp, totalSize=36.8 K 2023-03-28 20:56:25,020 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting f8c96febe200401293b4f722a3c55413, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1680036980384 2023-03-28 20:56:25,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting d3888ce4baeb4ab49ea4faf9a6a628d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1680036981521 2023-03-28 20:56:25,021 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] compactions.Compactor(207): Compacting 32480a2d11704652b25141c3cd7d4ed3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680036982634 2023-03-28 20:56:25,026 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] throttle.PressureAwareThroughputController(145): 1f0e2462e279947a10f94052c3152427#C#compaction#662 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-28 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/4c357fab78ec4a02a9059ef1edb4f8bf as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4c357fab78ec4a02a9059ef1edb4f8bf 2023-03-28 20:56:25,435 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/C of 1f0e2462e279947a10f94052c3152427 into 4c357fab78ec4a02a9059ef1edb4f8bf(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:25,435 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:25,435 INFO [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/C, priority=13, startTime=1680036985001; duration=0sec 2023-03-28 20:56:25,435 DEBUG [RS:0;jenkins-hbase4:42693-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:25,662 DEBUG [Thread-4128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52104714 to 127.0.0.1:56827 2023-03-28 20:56:25,662 DEBUG [Thread-4128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:25,664 DEBUG [Thread-4126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6257294f to 127.0.0.1:56827 2023-03-28 20:56:25,664 DEBUG [Thread-4126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:25,669 DEBUG [Thread-4132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:56827 2023-03-28 20:56:25,669 DEBUG [Thread-4132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:25,669 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5292 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5139 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5085 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5285 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(398): read 5148 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-28 20:56:25,670 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:56:25,671 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c0766f0 to 127.0.0.1:56827 2023-03-28 20:56:25,671 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:25,671 INFO [Listener at localhost/34739] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-28 20:56:25,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-28 20:56:25,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:25,674 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036985674"}]},"ts":"1680036985674"} 2023-03-28 20:56:25,675 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-28 20:56:25,676 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-28 20:56:25,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, UNASSIGN}] 2023-03-28 20:56:25,678 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, UNASSIGN 2023-03-28 20:56:25,679 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:25,679 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036985679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680036985679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680036985679"}]},"ts":"1680036985679"} 2023-03-28 20:56:25,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956}] 2023-03-28 20:56:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:25,825 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/fffcbed741144d848d4dca7c476732c2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/fffcbed741144d848d4dca7c476732c2 2023-03-28 20:56:25,829 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1f0e2462e279947a10f94052c3152427/A of 1f0e2462e279947a10f94052c3152427 into fffcbed741144d848d4dca7c476732c2(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-28 20:56:25,829 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:25,829 INFO [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427., storeName=1f0e2462e279947a10f94052c3152427/A, priority=13, startTime=1680036985000; duration=0sec 2023-03-28 20:56:25,829 DEBUG [RS:0;jenkins-hbase4:42693-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-28 20:56:25,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:25,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1f0e2462e279947a10f94052c3152427, disabling compactions & flushes 2023-03-28 20:56:25,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. after waiting 0 ms 2023-03-28 20:56:25,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:25,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1f0e2462e279947a10f94052c3152427 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=A 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=B 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1f0e2462e279947a10f94052c3152427, store=C 2023-03-28 20:56:25,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-28 20:56:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:26,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303288faa9d48a42c4dd2a2398d511b1c4f30 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288faa9d48a42c4dd2a2398d511b1c4f30 2023-03-28 20:56:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:26,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=749, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/cb1e15b4ab07405792de30f64002e386 2023-03-28 20:56:26,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:27,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f7759d290ff54d6295aa04d9c402a2d2 2023-03-28 20:56:27,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/d05c685e3eca491f862942f5f4c539e3 2023-03-28 20:56:27,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/A/cb1e15b4ab07405792de30f64002e386 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/cb1e15b4ab07405792de30f64002e386 2023-03-28 20:56:27,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/cb1e15b4ab07405792de30f64002e386, entries=100, sequenceid=749, filesize=18.6 K 2023-03-28 20:56:27,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/B/f7759d290ff54d6295aa04d9c402a2d2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f7759d290ff54d6295aa04d9c402a2d2 2023-03-28 20:56:27,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f7759d290ff54d6295aa04d9c402a2d2, entries=100, sequenceid=749, filesize=9.5 K 2023-03-28 20:56:27,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/.tmp/C/d05c685e3eca491f862942f5f4c539e3 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d05c685e3eca491f862942f5f4c539e3 2023-03-28 20:56:27,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d05c685e3eca491f862942f5f4c539e3, entries=100, sequenceid=749, filesize=9.5 K 2023-03-28 20:56:27,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 1f0e2462e279947a10f94052c3152427 in 1642ms, sequenceid=749, compaction requested=false 2023-03-28 20:56:27,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/c02f4c62e7ff46639cc2986912fd02d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e4b3fdfd52c743b9bccfd1da1b390175, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e1be5c961e1844afba8a2768145c63e9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b1eebae9d402443ab093a630eba73daa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d3851bfaadf44500a5f75ecf84f526ae, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/72f84f295f6140929e57a52b3aa95d31, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c600025df494d63886ebfbdc1b3dcd4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/088a500f8f774c6f881462255e2a413a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931] to archive 2023-03-28 20:56:27,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:27,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1b709773445d4b18ba7b6e3c6cdcdb6e 2023-03-28 20:56:27,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c34fd1c687246ffb555af6a4c24f25b 2023-03-28 20:56:27,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/c02f4c62e7ff46639cc2986912fd02d2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/c02f4c62e7ff46639cc2986912fd02d2 2023-03-28 20:56:27,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/2a02c11d603d4af68281096b124de5e0 2023-03-28 20:56:27,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/34b885675e374e578e3cead37332a90c 2023-03-28 20:56:27,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e4b3fdfd52c743b9bccfd1da1b390175 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e4b3fdfd52c743b9bccfd1da1b390175 2023-03-28 20:56:27,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9cd37a7fbd614503b45570dee8720c3c 2023-03-28 20:56:27,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b73c9ca65c6b45629ad5897e3a62bbc5 2023-03-28 20:56:27,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/549d54bf8eb7450ba2be864f1b6ba64d 2023-03-28 20:56:27,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1987eead72504e8fb2220c09803e7e39 2023-03-28 20:56:27,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e1be5c961e1844afba8a2768145c63e9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e1be5c961e1844afba8a2768145c63e9 2023-03-28 20:56:27,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/69f5e73143384613b03bf64cb4a88407 2023-03-28 20:56:27,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/bae6277df83b4980b7f42f3ec0ef9269 2023-03-28 20:56:27,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4043c8451a884496b439dac9ba5e992d 2023-03-28 20:56:27,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/33cb7d88de944b549b28db32cc694613 2023-03-28 20:56:27,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e68d9bcac60a4c26a578f291d812b372 2023-03-28 20:56:27,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/7e5076b9c3514eec8fa9dddf84b22157 2023-03-28 20:56:27,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/9efca908c5e049f68bb1d1f67cd8b205 2023-03-28 20:56:27,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b1eebae9d402443ab093a630eba73daa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/b1eebae9d402443ab093a630eba73daa 2023-03-28 20:56:27,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/58baa96990d4437980a3ccf765f6b382 2023-03-28 20:56:27,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d523db1102744fff8b818bde297b3d0c 2023-03-28 20:56:27,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d3851bfaadf44500a5f75ecf84f526ae to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/d3851bfaadf44500a5f75ecf84f526ae 2023-03-28 20:56:27,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/5e61af4c55d649e193133df9ffe6afba 2023-03-28 20:56:27,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/72f84f295f6140929e57a52b3aa95d31 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/72f84f295f6140929e57a52b3aa95d31 2023-03-28 20:56:27,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/84e542a85a23408496896f5a3fdcc803 2023-03-28 20:56:27,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/e15a808954984fa893f06020c9d7a4e4 2023-03-28 20:56:27,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/28ecdd5330264f5187d282f542aff5af 2023-03-28 20:56:27,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/146370775ecf411794cd09e360851f4f 2023-03-28 20:56:27,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c600025df494d63886ebfbdc1b3dcd4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/4c600025df494d63886ebfbdc1b3dcd4 2023-03-28 20:56:27,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/1677833a3f004a41998c4783d123e65f 2023-03-28 20:56:27,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/44eb69e5a4b4477b833f0ffef5230161 2023-03-28 20:56:27,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/aa5fd9eb47e548f08521329b4521b9a6 2023-03-28 20:56:27,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/088a500f8f774c6f881462255e2a413a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/088a500f8f774c6f881462255e2a413a 2023-03-28 20:56:27,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/48a9333d271542a885c2a765d4eaca63 2023-03-28 20:56:27,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/885be9a4cfe54a789128059cb192f931 2023-03-28 20:56:27,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b70bc470870f40c899aebe91a1a1c20f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f28ce113aa1147a389aeab60417793c5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5952d8f0e433486992fc92822aea7a4a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/21598a97fcc147eaae64397108239a3a, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/296f5a2cfddf4552875963feb62ae7de, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/fd2d8e7cd5e14af48c1f65923f9e47dc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/ac5594fdffc3400686561b7dd83ae5e8, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/72a5dc74e7be4965b4bab39ae14aff9b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da] to archive 2023-03-28 20:56:27,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:27,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3c062dd39da549bda10c8a19f217da89 2023-03-28 20:56:27,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/51bc650255b4468f8b195c0f42e9c726 2023-03-28 20:56:27,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b70bc470870f40c899aebe91a1a1c20f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b70bc470870f40c899aebe91a1a1c20f 2023-03-28 20:56:27,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9e658e981902415b8d9e59992c8802aa 2023-03-28 20:56:27,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6ad91565a0ce4052b6f18fc38db8dc1b 2023-03-28 20:56:27,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f28ce113aa1147a389aeab60417793c5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f28ce113aa1147a389aeab60417793c5 2023-03-28 20:56:27,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/11f291b714e04c39a05616e4ce0f8242 2023-03-28 20:56:27,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/bca7d690824a4d3185d1d3c21a7af274 2023-03-28 20:56:27,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f2c822d0bfdb4caba9472c7847b14d06 2023-03-28 20:56:27,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/6994cbe1506c44a6a71416168e8f847a 2023-03-28 20:56:27,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5952d8f0e433486992fc92822aea7a4a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5952d8f0e433486992fc92822aea7a4a 2023-03-28 20:56:27,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3676aa396e7c49db87278be5b614a2e2 2023-03-28 20:56:27,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/7268b624ce384033884bbe9d5dcad31a 2023-03-28 20:56:27,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e63d14e21ae847da90a19aaa071aec1c 2023-03-28 20:56:27,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/28d171aea4ea4996ba1bba7f4cb9725b 2023-03-28 20:56:27,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/95790a13384b41b4b710f6e3ff9f8cf9 2023-03-28 20:56:27,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f0c684b71c2b4d449cdf842164b539dc 2023-03-28 20:56:27,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/21598a97fcc147eaae64397108239a3a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/21598a97fcc147eaae64397108239a3a 2023-03-28 20:56:27,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/5df85744151a41bc9342e74b578e36c8 2023-03-28 20:56:27,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/b32c60cbf3af41688506133cd97f14e1 2023-03-28 20:56:27,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/296f5a2cfddf4552875963feb62ae7de to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/296f5a2cfddf4552875963feb62ae7de 2023-03-28 20:56:27,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0b57555dc2de48e19417952c1a5a9e91 2023-03-28 20:56:27,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2956e8a771444a19838d4886b84c6112 2023-03-28 20:56:27,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/fd2d8e7cd5e14af48c1f65923f9e47dc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/fd2d8e7cd5e14af48c1f65923f9e47dc 2023-03-28 20:56:27,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0c6ae91edb5a422984eed980c3ed998b 2023-03-28 20:56:27,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/331276eb3393463cad7e15227318bb96 2023-03-28 20:56:27,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/d487cbae9ac64ff28c25c4f0e874184c 2023-03-28 20:56:27,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/ac5594fdffc3400686561b7dd83ae5e8 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/ac5594fdffc3400686561b7dd83ae5e8 2023-03-28 20:56:27,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/2326380e2756453d9df0658371c914e5 2023-03-28 20:56:27,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/1862a3ed2d8f493498c87aac9c1e1206 2023-03-28 20:56:27,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/0d7678fbf3c84e328f0fc04dffbab4bb 2023-03-28 20:56:27,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/72a5dc74e7be4965b4bab39ae14aff9b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/72a5dc74e7be4965b4bab39ae14aff9b 2023-03-28 20:56:27,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/e3bae6f2f20341a7b2234b2c1157b6e1 2023-03-28 20:56:27,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/3129baf7cff7454bab4094825628d4e0 2023-03-28 20:56:27,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/39d5636837f74102b048ab36f868c9da 2023-03-28 20:56:27,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/54cded68b6874625b34caa901105e4be, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b24d570d2d734d1dbfd1898bb5ee1104, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/66c914b3a8fb4bf5aa5f78b3cb842960, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c70f7fe5cfb542179f0e968eedb08063, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/688d1906d8a54d288c32ae9c87f0f957, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/605eec9d048344fc94f4c38299124d95, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f8c96febe200401293b4f722a3c55413, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3] to archive 2023-03-28 20:56:27,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-28 20:56:27,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1395e3575e9a43ca9c1ed098561b9db5 2023-03-28 20:56:27,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/366044011eb547308b2b808cbc9d2319 2023-03-28 20:56:27,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/54cded68b6874625b34caa901105e4be to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/54cded68b6874625b34caa901105e4be 2023-03-28 20:56:27,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f108047291204698ad190e7d5a37cd52 2023-03-28 20:56:27,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e9527b29ef8149ce81131aa7dba01714 2023-03-28 20:56:27,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b24d570d2d734d1dbfd1898bb5ee1104 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b24d570d2d734d1dbfd1898bb5ee1104 2023-03-28 20:56:27,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f5683deab72c4e1a83e5f53f7523c2f4 2023-03-28 20:56:27,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a2ec3ff90900482eb89aabb93c6c0c13 2023-03-28 20:56:27,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/9c7c0c7f1ab1427097499f29caf7bdd0 2023-03-28 20:56:27,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/b165cfd346f142a9adc084ad38240321 2023-03-28 20:56:27,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/66c914b3a8fb4bf5aa5f78b3cb842960 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/66c914b3a8fb4bf5aa5f78b3cb842960 2023-03-28 20:56:27,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c064323e615a4ab088e70de72085bb6b 2023-03-28 20:56:27,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/1763992fe52e4155b144eafbefe17887 2023-03-28 20:56:27,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/92193ad7c9e644cea5e280452df5be40 2023-03-28 20:56:27,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/cd929509cbe24fbb8d9dc467a5bb9bce 2023-03-28 20:56:27,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/e5be5ac46dca4e82ac448bb85f99b658 2023-03-28 20:56:27,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/80bc338d20cf4c9491dd9689f99580ea 2023-03-28 20:56:27,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ae8ac9ddb2cb40ae8de02e7c03c44cdc 2023-03-28 20:56:27,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/3be552f2b5a24cdbad3aa5d785c7cebb 2023-03-28 20:56:27,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5c5ca5668e7341dc81a42b7ef9d7583c 2023-03-28 20:56:27,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c70f7fe5cfb542179f0e968eedb08063 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/c70f7fe5cfb542179f0e968eedb08063 2023-03-28 20:56:27,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f6b8eba24de848e5bbadb34e3f210971 2023-03-28 20:56:27,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/12f3de1272e74979b9ea188cd493965f 2023-03-28 20:56:27,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/688d1906d8a54d288c32ae9c87f0f957 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/688d1906d8a54d288c32ae9c87f0f957 2023-03-28 20:56:27,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/5eeed48d9c35460db5282215a9feaaeb 2023-03-28 20:56:27,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f0c84b9bd9ec49109d407e7611fc0679 2023-03-28 20:56:27,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/a3a0080f4ee84034bb9e7d5d56abc348 2023-03-28 20:56:27,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/605eec9d048344fc94f4c38299124d95 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/605eec9d048344fc94f4c38299124d95 2023-03-28 20:56:27,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/ad9e836a633b43aa839efd73b82ad76c 2023-03-28 20:56:27,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/2b8daf9133c244bf97d5c2768ffa378c 2023-03-28 20:56:27,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/52cb47f6274948238fef6736ed648344 2023-03-28 20:56:27,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f8c96febe200401293b4f722a3c55413 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/f8c96febe200401293b4f722a3c55413 2023-03-28 20:56:27,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4ecce0b2991c45b0ab22b998890d24c9 2023-03-28 20:56:27,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d3888ce4baeb4ab49ea4faf9a6a628d2 2023-03-28 20:56:27,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/32480a2d11704652b25141c3cd7d4ed3 2023-03-28 20:56:27,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits/752.seqid, newMaxSeqId=752, maxSeqId=4 2023-03-28 20:56:27,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427. 2023-03-28 20:56:27,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1f0e2462e279947a10f94052c3152427: 2023-03-28 20:56:27,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:27,598 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1f0e2462e279947a10f94052c3152427, regionState=CLOSED 2023-03-28 20:56:27,598 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680036987598"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680036987598"}]},"ts":"1680036987598"} 2023-03-28 20:56:27,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-28 20:56:27,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 1f0e2462e279947a10f94052c3152427, server=jenkins-hbase4.apache.org,42693,1680036815956 in 1.9190 sec 2023-03-28 20:56:27,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-28 20:56:27,601 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1f0e2462e279947a10f94052c3152427, UNASSIGN in 1.9230 sec 2023-03-28 20:56:27,602 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680036987602"}]},"ts":"1680036987602"} 2023-03-28 20:56:27,603 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-28 20:56:27,604 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-28 20:56:27,605 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9330 sec 2023-03-28 20:56:27,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-28 20:56:27,779 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-28 20:56:27,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-28 20:56:27,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,782 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-28 20:56:27,783 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,787 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:27,789 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C, FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits] 2023-03-28 20:56:27,791 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/cb1e15b4ab07405792de30f64002e386 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/cb1e15b4ab07405792de30f64002e386 2023-03-28 20:56:27,792 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/fffcbed741144d848d4dca7c476732c2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/A/fffcbed741144d848d4dca7c476732c2 2023-03-28 20:56:27,794 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9222b7c30b34433a84132e519de89b82 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/9222b7c30b34433a84132e519de89b82 2023-03-28 20:56:27,795 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f7759d290ff54d6295aa04d9c402a2d2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/B/f7759d290ff54d6295aa04d9c402a2d2 2023-03-28 20:56:27,798 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4c357fab78ec4a02a9059ef1edb4f8bf to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/4c357fab78ec4a02a9059ef1edb4f8bf 2023-03-28 20:56:27,799 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d05c685e3eca491f862942f5f4c539e3 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/C/d05c685e3eca491f862942f5f4c539e3 2023-03-28 20:56:27,801 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits/752.seqid to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427/recovered.edits/752.seqid 2023-03-28 20:56:27,801 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/.tmp/data/default/TestAcidGuarantees/1f0e2462e279947a10f94052c3152427 2023-03-28 20:56:27,801 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-28 20:56:27,802 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:56:27,802 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-28 20:56:27,804 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328174660d23ff54a8495550ce51e9cde73 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328174660d23ff54a8495550ce51e9cde73 2023-03-28 20:56:27,805 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032819dcd1e0d226404e89dba211d512a6e7 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032819dcd1e0d226404e89dba211d512a6e7 2023-03-28 20:56:27,806 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328239c8fa7c8bd486fa1ef0861fb04606a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328239c8fa7c8bd486fa1ef0861fb04606a 2023-03-28 20:56:27,806 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032826c16fbf3e41432191d74f9cdeea2c21 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032826c16fbf3e41432191d74f9cdeea2c21 2023-03-28 20:56:27,807 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282cfb568fd1094de4b23dcf318f697f6d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282cfb568fd1094de4b23dcf318f697f6d 2023-03-28 20:56:27,808 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282ddaae16bcef4196bcb9d891c7a24008 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303282ddaae16bcef4196bcb9d891c7a24008 2023-03-28 20:56:27,809 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328336f826ab43e412da370ff9bade360d1 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328336f826ab43e412da370ff9bade360d1 2023-03-28 20:56:27,810 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303283738966fd9184a129b9936bc0b1d5639 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303283738966fd9184a129b9936bc0b1d5639 2023-03-28 20:56:27,811 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032859eb3157377f418aaabaa64f7f7ee89a to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032859eb3157377f418aaabaa64f7f7ee89a 2023-03-28 20:56:27,812 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285d5936f9e4514a4fac1663eab7614bfa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285d5936f9e4514a4fac1663eab7614bfa 2023-03-28 20:56:27,812 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e4d411bf70340ecb9ed0286b7c50ada to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e4d411bf70340ecb9ed0286b7c50ada 2023-03-28 20:56:27,813 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e848e2eb5214f64a43557b4e8adbf73 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303285e848e2eb5214f64a43557b4e8adbf73 2023-03-28 20:56:27,814 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286029ec8d74f34ed3b2388ddca1f6a04d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303286029ec8d74f34ed3b2388ddca1f6a04d 2023-03-28 20:56:27,815 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328726f31594bfe4f879f0134a941c9e6fa to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328726f31594bfe4f879f0134a941c9e6fa 2023-03-28 20:56:27,816 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032874904c334b9f4a3cadc774670fda3e45 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032874904c334b9f4a3cadc774670fda3e45 2023-03-28 20:56:27,816 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875eb5096b5314d6a93ac522bb44caaa0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032875eb5096b5314d6a93ac522bb44caaa0 2023-03-28 20:56:27,817 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287879ec273f964082aff23aa1a5e5031e to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287879ec273f964082aff23aa1a5e5031e 2023-03-28 20:56:27,818 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287fe6efddf5554d2f9df2f9de7b997523 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303287fe6efddf5554d2f9df2f9de7b997523 2023-03-28 20:56:27,819 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032886312aa4c6044e57801b2d436b9907f2 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032886312aa4c6044e57801b2d436b9907f2 2023-03-28 20:56:27,820 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288faa9d48a42c4dd2a2398d511b1c4f30 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303288faa9d48a42c4dd2a2398d511b1c4f30 2023-03-28 20:56:27,820 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289a5c0a8705f643edae3e4f6f6bbbaff0 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303289a5c0a8705f643edae3e4f6f6bbbaff0 2023-03-28 20:56:27,821 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328af7c5ecaaf324f9aa6c6d041bf4c7a0d to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328af7c5ecaaf324f9aa6c6d041bf4c7a0d 2023-03-28 20:56:27,822 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328bc08bc122e894dd1b8cee77212c9b693 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328bc08bc122e894dd1b8cee77212c9b693 2023-03-28 20:56:27,823 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb48348fc6da4a07af4e7943959873e4 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328cb48348fc6da4a07af4e7943959873e4 2023-03-28 20:56:27,823 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ea14b168476e46a88e114958b16a19dd to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328ea14b168476e46a88e114958b16a19dd 2023-03-28 20:56:27,824 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f294b2436484478788eb79cf1c1d5956 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f294b2436484478788eb79cf1c1d5956 2023-03-28 20:56:27,825 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f470450766dc43aa9576370793fc6b10 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328f470450766dc43aa9576370793fc6b10 2023-03-28 20:56:27,826 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328fb069533665b426297aadd5ab7234938 to hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230328fb069533665b426297aadd5ab7234938 2023-03-28 20:56:27,826 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-28 20:56:27,828 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,829 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-28 20:56:27,831 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-28 20:56:27,832 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,832 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-28 20:56:27,832 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680036987832"}]},"ts":"9223372036854775807"} 2023-03-28 20:56:27,833 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-28 20:56:27,833 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1f0e2462e279947a10f94052c3152427, NAME => 'TestAcidGuarantees,,1680036960625.1f0e2462e279947a10f94052c3152427.', STARTKEY => '', ENDKEY => ''}] 2023-03-28 20:56:27,833 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-28 20:56:27,833 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680036987833"}]},"ts":"9223372036854775807"} 2023-03-28 20:56:27,834 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-28 20:56:27,836 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-28 20:56:27,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-03-28 20:56:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44521] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-28 20:56:27,883 INFO [Listener at localhost/34739] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-28 20:56:27,895 INFO [Listener at localhost/34739] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 232), OpenFileDescriptor=594 (was 583) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=420 (was 386) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7171 (was 6941) - AvailableMemoryMB LEAK? - 2023-03-28 20:56:27,895 INFO [Listener at localhost/34739] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-28 20:56:27,895 INFO [Listener at localhost/34739] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-28 20:56:27,896 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d205425 to 127.0.0.1:56827 2023-03-28 20:56:27,896 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:27,896 DEBUG [Listener at localhost/34739] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-28 20:56:27,896 DEBUG [Listener at localhost/34739] util.JVMClusterUtil(257): Found active master hash=1493635499, stopped=false 2023-03-28 20:56:27,896 INFO [Listener at localhost/34739] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:56:27,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-28 20:56:27,898 INFO [Listener at localhost/34739] procedure2.ProcedureExecutor(629): Stopping 2023-03-28 20:56:27,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-28 20:56:27,898 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:56:27,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-28 20:56:27,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-28 20:56:27,899 DEBUG [Listener at localhost/34739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x063e7276 to 127.0.0.1:56827 2023-03-28 20:56:27,899 DEBUG [Listener at localhost/34739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:27,899 INFO [Listener at localhost/34739] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42693,1680036815956' ***** 2023-03-28 20:56:27,899 INFO [Listener at localhost/34739] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-28 20:56:27,900 INFO [RS:0;jenkins-hbase4:42693] regionserver.HeapMemoryManager(220): Stopping 2023-03-28 20:56:27,900 INFO [RS:0;jenkins-hbase4:42693] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-28 20:56:27,900 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-28 20:56:27,900 INFO [RS:0;jenkins-hbase4:42693] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-28 20:56:27,900 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(3304): Received CLOSE for 097dd7e52a34fa343db86f0dfd060f36 2023-03-28 20:56:27,900 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:27,900 DEBUG [RS:0;jenkins-hbase4:42693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x086c7f07 to 127.0.0.1:56827 2023-03-28 20:56:27,901 DEBUG [RS:0;jenkins-hbase4:42693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:27,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 097dd7e52a34fa343db86f0dfd060f36, disabling compactions & flushes 2023-03-28 20:56:27,901 INFO [RS:0;jenkins-hbase4:42693] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-28 20:56:27,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:56:27,901 INFO [RS:0;jenkins-hbase4:42693] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-28 20:56:27,901 INFO [RS:0;jenkins-hbase4:42693] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-28 20:56:27,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:56:27,901 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-28 20:56:27,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. after waiting 0 ms 2023-03-28 20:56:27,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:56:27,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 097dd7e52a34fa343db86f0dfd060f36 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-28 20:56:27,902 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-28 20:56:27,902 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 097dd7e52a34fa343db86f0dfd060f36=hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36.} 2023-03-28 20:56:27,902 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-28 20:56:27,902 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-28 20:56:27,902 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-28 20:56:27,902 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-28 20:56:27,902 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-28 20:56:27,903 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-28 20:56:27,904 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1505): Waiting on 097dd7e52a34fa343db86f0dfd060f36, 1588230740 2023-03-28 20:56:27,914 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/info/bce42a5cab5f4892a0b1d6aa8ddf4902 2023-03-28 20:56:27,916 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-28 20:56:27,918 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bce42a5cab5f4892a0b1d6aa8ddf4902 2023-03-28 20:56:28,104 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1505): Waiting on 097dd7e52a34fa343db86f0dfd060f36, 1588230740 2023-03-28 20:56:28,304 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1505): Waiting on 097dd7e52a34fa343db86f0dfd060f36, 1588230740 2023-03-28 20:56:28,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/.tmp/info/21d41d4965a04f059fc537c3a940e220 2023-03-28 20:56:28,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/.tmp/info/21d41d4965a04f059fc537c3a940e220 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/info/21d41d4965a04f059fc537c3a940e220 2023-03-28 20:56:28,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/info/21d41d4965a04f059fc537c3a940e220, entries=2, sequenceid=6, filesize=4.8 K 2023-03-28 20:56:28,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 097dd7e52a34fa343db86f0dfd060f36 in 421ms, sequenceid=6, compaction requested=false 2023-03-28 20:56:28,324 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/rep_barrier/224c01aac0e540308738d2425ecc35f2 2023-03-28 20:56:28,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/namespace/097dd7e52a34fa343db86f0dfd060f36/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-28 20:56:28,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 224c01aac0e540308738d2425ecc35f2 2023-03-28 20:56:28,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:56:28,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 097dd7e52a34fa343db86f0dfd060f36: 2023-03-28 20:56:28,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680036818222.097dd7e52a34fa343db86f0dfd060f36. 2023-03-28 20:56:28,505 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-28 20:56:28,705 DEBUG [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-28 20:56:28,720 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-28 20:56:28,720 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-28 20:56:28,738 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/table/849c4bf437ab4129a65130488f9b807c 2023-03-28 20:56:28,741 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 849c4bf437ab4129a65130488f9b807c 2023-03-28 20:56:28,741 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/info/bce42a5cab5f4892a0b1d6aa8ddf4902 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/info/bce42a5cab5f4892a0b1d6aa8ddf4902 2023-03-28 20:56:28,744 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bce42a5cab5f4892a0b1d6aa8ddf4902 2023-03-28 20:56:28,744 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/info/bce42a5cab5f4892a0b1d6aa8ddf4902, entries=22, sequenceid=93, filesize=7.1 K 2023-03-28 20:56:28,745 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/rep_barrier/224c01aac0e540308738d2425ecc35f2 as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/rep_barrier/224c01aac0e540308738d2425ecc35f2 2023-03-28 20:56:28,748 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 224c01aac0e540308738d2425ecc35f2 2023-03-28 20:56:28,748 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/rep_barrier/224c01aac0e540308738d2425ecc35f2, entries=6, sequenceid=93, filesize=5.5 K 2023-03-28 20:56:28,749 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/.tmp/table/849c4bf437ab4129a65130488f9b807c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/table/849c4bf437ab4129a65130488f9b807c 2023-03-28 20:56:28,751 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 849c4bf437ab4129a65130488f9b807c 2023-03-28 20:56:28,751 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/table/849c4bf437ab4129a65130488f9b807c, entries=9, sequenceid=93, filesize=5.5 K 2023-03-28 20:56:28,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 850ms, sequenceid=93, compaction requested=false 2023-03-28 20:56:28,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-28 20:56:28,756 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-28 20:56:28,757 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-28 20:56:28,757 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-28 20:56:28,757 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-28 20:56:28,905 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42693,1680036815956; all regions closed. 2023-03-28 20:56:28,913 DEBUG [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/oldWALs 2023-03-28 20:56:28,914 INFO [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42693%2C1680036815956.meta:.meta(num 1680036817980) 2023-03-28 20:56:28,918 DEBUG [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/oldWALs 2023-03-28 20:56:28,918 INFO [RS:0;jenkins-hbase4:42693] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42693%2C1680036815956:(num 1680036817833) 2023-03-28 20:56:28,918 DEBUG [RS:0;jenkins-hbase4:42693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:28,918 INFO [RS:0;jenkins-hbase4:42693] regionserver.LeaseManager(133): Closed leases 2023-03-28 20:56:28,918 INFO [RS:0;jenkins-hbase4:42693] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-28 20:56:28,918 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-28 20:56:28,920 INFO [RS:0;jenkins-hbase4:42693] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42693 2023-03-28 20:56:28,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42693,1680036815956 2023-03-28 20:56:28,926 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-28 20:56:28,927 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20c5d579 rejected from java.util.concurrent.ThreadPoolExecutor@24fa45b9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2607] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:28,927 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-28 20:56:28,927 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5eb3d1d2 rejected from java.util.concurrent.ThreadPoolExecutor@24fa45b9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2607] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:28,928 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42693,1680036815956] 2023-03-28 20:56:28,928 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42693,1680036815956; numProcessing=1 2023-03-28 20:56:28,930 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42693,1680036815956 already deleted, retry=false 2023-03-28 20:56:28,930 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42693,1680036815956 expired; onlineServers=0 2023-03-28 20:56:28,930 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44521,1680036814691' ***** 2023-03-28 20:56:28,930 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-28 20:56:28,930 DEBUG [M:0;jenkins-hbase4:44521] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@183a1b90, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-28 20:56:28,930 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44521,1680036814691 2023-03-28 20:56:28,930 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44521,1680036814691; all regions closed. 2023-03-28 20:56:28,930 DEBUG [M:0;jenkins-hbase4:44521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-28 20:56:28,930 DEBUG [M:0;jenkins-hbase4:44521] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-28 20:56:28,931 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-28 20:56:28,931 DEBUG [M:0;jenkins-hbase4:44521] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-28 20:56:28,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680036817492] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680036817492,5,FailOnTimeoutGroup] 2023-03-28 20:56:28,931 INFO [M:0;jenkins-hbase4:44521] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-28 20:56:28,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680036817492] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680036817492,5,FailOnTimeoutGroup] 2023-03-28 20:56:28,931 INFO [M:0;jenkins-hbase4:44521] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-28 20:56:28,931 INFO [M:0;jenkins-hbase4:44521] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-28 20:56:28,932 DEBUG [M:0;jenkins-hbase4:44521] master.HMaster(1512): Stopping service threads 2023-03-28 20:56:28,932 INFO [M:0;jenkins-hbase4:44521] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-28 20:56:28,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-28 20:56:28,932 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-28 20:56:28,932 ERROR [M:0;jenkins-hbase4:44521] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1683253216) connection to localhost/127.0.0.1:34423 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-28 20:56:28,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-28 20:56:28,933 INFO [M:0;jenkins-hbase4:44521] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-28 20:56:28,933 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-28 20:56:28,933 DEBUG [M:0;jenkins-hbase4:44521] zookeeper.ZKUtil(398): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-28 20:56:28,933 WARN [M:0;jenkins-hbase4:44521] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-28 20:56:28,933 INFO [M:0;jenkins-hbase4:44521] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-28 20:56:28,933 INFO [M:0;jenkins-hbase4:44521] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-28 20:56:28,934 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-28 20:56:28,934 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:56:28,934 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:56:28,934 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-28 20:56:28,934 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:56:28,934 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.05 KB heapSize=393.73 KB 2023-03-28 20:56:28,946 INFO [M:0;jenkins-hbase4:44521] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.05 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/38a663faa32840e78fba09738938c35c 2023-03-28 20:56:28,949 INFO [M:0;jenkins-hbase4:44521] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 38a663faa32840e78fba09738938c35c 2023-03-28 20:56:28,950 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/38a663faa32840e78fba09738938c35c as hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/38a663faa32840e78fba09738938c35c 2023-03-28 20:56:28,952 INFO [M:0;jenkins-hbase4:44521] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 38a663faa32840e78fba09738938c35c 2023-03-28 20:56:28,952 INFO [M:0;jenkins-hbase4:44521] regionserver.HStore(1080): Added hdfs://localhost:34423/user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/38a663faa32840e78fba09738938c35c, entries=65, sequenceid=578, filesize=24.7 K 2023-03-28 20:56:28,953 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegion(2948): Finished flush of dataSize ~345.05 KB/353333, heapSize ~393.72 KB/403168, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=578, compaction requested=false 2023-03-28 20:56:28,954 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-28 20:56:28,954 DEBUG [M:0;jenkins-hbase4:44521] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-28 20:56:28,957 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/161e7b7b-7109-c185-807b-2e6c3944f4ec/MasterData/WALs/jenkins-hbase4.apache.org,44521,1680036814691/jenkins-hbase4.apache.org%2C44521%2C1680036814691.1680036816797 not finished, retry = 0 2023-03-28 20:56:29,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-28 20:56:29,029 INFO [RS:0;jenkins-hbase4:42693] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42693,1680036815956; zookeeper connection closed. 2023-03-28 20:56:29,029 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26f2e95e rejected from java.util.concurrent.ThreadPoolExecutor@24fa45b9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2607] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:29,029 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): regionserver:42693-0x1012ce745ce0001, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-28 20:56:29,029 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41aa5ca4 rejected from java.util.concurrent.ThreadPoolExecutor@24fa45b9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2607] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:29,029 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@a3a8f49] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@a3a8f49 2023-03-28 20:56:29,030 INFO [Listener at localhost/34739] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-28 20:56:29,058 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-28 20:56:29,058 INFO [M:0;jenkins-hbase4:44521] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-28 20:56:29,059 INFO [M:0;jenkins-hbase4:44521] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44521 2023-03-28 20:56:29,062 DEBUG [M:0;jenkins-hbase4:44521] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44521,1680036814691 already deleted, retry=false 2023-03-28 20:56:29,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-28 20:56:29,163 INFO [M:0;jenkins-hbase4:44521] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44521,1680036814691; zookeeper connection closed. 2023-03-28 20:56:29,163 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64015780 rejected from java.util.concurrent.ThreadPoolExecutor@13968c2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4352] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:29,163 DEBUG [Listener at localhost/34739-EventThread] zookeeper.ZKWatcher(600): master:44521-0x1012ce745ce0000, quorum=127.0.0.1:56827, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-28 20:56:29,163 ERROR [Listener at localhost/34739-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4191f8d9 rejected from java.util.concurrent.ThreadPoolExecutor@13968c2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4352] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-28 20:56:29,165 WARN [Listener at localhost/34739] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-28 20:56:29,169 INFO [Listener at localhost/34739] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-28 20:56:29,286 WARN [BP-255062507-172.31.14.131-1680036811406 heartbeating to localhost/127.0.0.1:34423] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-28 20:56:29,286 WARN [BP-255062507-172.31.14.131-1680036811406 heartbeating to localhost/127.0.0.1:34423] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-255062507-172.31.14.131-1680036811406 (Datanode Uuid e8b38ec5-35e4-41ef-a012-5de0dd603912) service to localhost/127.0.0.1:34423 2023-03-28 20:56:29,289 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f/dfs/data/data1/current/BP-255062507-172.31.14.131-1680036811406] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-28 20:56:29,290 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eeeeb1b5-95f7-75e3-5570-a91eebbbfeda/cluster_db6382a1-f5a8-e25b-d3a8-92da1cc8d38f/dfs/data/data2/current/BP-255062507-172.31.14.131-1680036811406] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-28 20:56:29,332 INFO [Listener at localhost/34739] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-28 20:56:29,505 INFO [Listener at localhost/34739] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-28 20:56:29,697 INFO [Listener at localhost/34739] hbase.HBaseTestingUtility(1293): Minicluster is down